Project 31-A

As I continue to program and experiment with C#/.Net technologies I develop new “favorite” ways of doing things. This is my current iteration of the perfect call to SQL Server. I use this style of method in middle layer classes that are called from ASP.NET pages and need access to SQL Server.  This style of method is built on many things, including a good understand of how SQL server works, real life performance and scalability testing, and the new fondness for yield return. public IEnumerable<Member> Members() { using (SqlConnection sqlConnection = new SqlConnection( ConfigurationManager.ConnectionStrings[ "connectionstring" ].ConnectionString)) { sqlConnection.Open(); StringBuilder sql = new StringBuilder(); sql.Append( "SELECT memberId\r\n" ); sql.Append( "FROM tbMembers\r\n" ); sql.Append( "WHERE deleted = 0 AND companyId = @CompanyId\r\n" ); ...

For a short period in my professional life I programmed computers before Internet access and search engines.  Trying to explain what this was like to my younger peers who have always been able to “Google” any programming problem and come up with an immediate answer is impossible, even to comprehend how I did it is very hard for myself.  I relied on the senior programmers I was working with, a large code base to search for example code, a few books that littered my desk, and newsgroups and email lists.  Imagine that you have to solve every problem yourself without being able to type a few key phrases into Google and copy and paste example code from a helpful blog directly into your code base.  To our advantage at that time we were working at a much lower level in the code, with less layers of abstractions (except for a handful of Windows controls).   Google is a powerful tool that makes my job much easier, in fact after Visual Studio it is my number two tool ...

Last year I had a minor project that was importing a divergent collection of data into SQL Server. The typical file had thousands of records, on occasion million of records that were batch-sequence sensitive. A common C# pattern is to read one record and insert that into SQL Server. This pattern is not robust when the insert program dies unexpectedly because it may insert duplicate records (You can start adding widgets but that is not a KISS solution.).  SQL Utilities such as BCP etc, tend to require a lot of fiddling to get the import right (and tend not to be very tolerant of data changes).   The problem was further complicated because there was a variety of uploading machines (isolated domains) which was difficult to update with new code (secured controls machines). In general, there were folders for data to be dropped which were monitored by Windows Services that did the upload.   The pattern that I ended up using was simple: Convert each file to Xml in ...

In my earlier blog, we found that arrays were twice as fast as IEnumerables. So how do you get data into an array with good performance?   This note looks at how to get a stream of objects into an array with the best performance. Using Object Browser in Visual Studio quickly finds all of the built in ToArray() methods. The other alternative is to create a fix size array and expand it as needed (we will use 1000 items as the default and expand in 1000 items chunks) – this was the worst performer.   The numbers for x86 compiled code is shown below: Row Labels Min of Msec ArrayResize<> 198.24 HashSet<> 69.34 IEnumerable 27.34 List<> 14.65 NoOp 0.00 Queue<> 17.58 Stack<> 21.48   And doing it for explicit x64 compiled code fo...

I have often changed fornext statements to lambda notation when prompted by code analysis tools. Every time that I did the change, I asked myself “I need to verify that this is the correct choice in terms of performance”. I assumed that it is because Microsoft had been trying to improve performance (which it did with XDocument being twice as fast as XmlDocument, which I documented in an earlier blog ). I created a simple collection of different ways of walking an enumeration, with some very interesting results.   Compiled for x86 explicitly Method Best Time x86 DATA   No Op   1.95 0 0 0 0 Ienumerate: foreach/var 16.6 16.6 17.58 19.53 18.55 16.6 Ienumerate: foreach/typed 16.6 17.6 18.55 18.55 16.6 16.6 ...

In my earlier blog I had summarized from experience and material on the web that was born out by my own observations. I decided to construct a very simple demo program as shown at the bottom of this blog. I loaded the same file with each approach and counted the nodes in a Xml file. The box was quad-x64 core with 12 Gigs of ram.   The results are very information for guidance. x86 Memory (K) Msec Memory Delta No Op 3196 1.95   XmlDocument 43912 1221.70 40716 XDocument 34700 836.90 31504 XmlReader 3696 299.80 500 x 64 Memory (K) Msec Memory Delta No Op 3184 0.98   XmlDocument 74900 1076.20 71716 XDocument 48252 ...

With AspNet, there are three possible paths to handling XML. There is a nice performance comparison done by Joe Ferner which augments many earlier studies between XmlDocument and XmlReader/XmlWriter. The use of XmlDocument  could be justified for quick and dirty prototyping because: It is often the fastest to code and debug It is suitable for neophytes to Xml For anything that is production --- forget XmlDocument!!!   It is a memory hog and slow! A quote in the article states it all “It is not surprising the XmlReader code is faster since both the XML DOM (XmlDocument) and LINQ to XML use XmlReader to do their reading.”   So if any volume of Xml is used, there is no rationale choice except for XmlReader/XmlWriter. I have processed gigabyte Xml files with these and have not seen memory usage go over 200K; on the other hand, I have max memory out with XmlDocument.   So for small pieces of data what do you use?  Coding and reading XmlReader/X...