|
Hi Guys!
I have following problem: I'm loading a XML File into a Dataset, well actually I pick up some data from the original File, modify it and save the objects to an arraylist. It works like a charm and does what it's supposed to do, but it's horribly slow.
The XML Files I work with have roughly 22000 seperate nodes and are about 50mb.
I'm looping through the whole Dataset with a foreach statement, which consumes the overwhelming time of the applications' runtime :)
Has anyone an idea how i could speed things up a little bit?
Tia
InTo$h - Moved bynobugzMVP, ModeratorFriday, August 07, 2009 12:23 AM (From:.NET Base Class Library)
-
|
| InTo$h Thursday, August 06, 2009 12:41 PM |
I would look into the parallel extensions framework and specifically the Parallel.ForEach method: http://www.microsoft.com/downloads/details.aspx?FamilyId=348F73FD-593D-4B3C-B055-694C50D2B0F3&displaylang=en Simplifies multithreading quite a bit. And check this out: http://blogs.msdn.com/pfxteam/
Jerry Schulist
Please remember to mark replies which answer your question as answers and vote for replies which are helpful. - Marked As Answer byLing WangMSFT, ModeratorThursday, August 13, 2009 3:19 AM
-
|
| inetscan Thursday, August 06, 2009 3:19 PM |
Hi, i would take a look at the XMLReader instead of using a dataset. You can do it like this:
Public Sub XMLReader(ByVal sFileName As String, ByVal QueryNode As String, ByVal QueryAttribute As String)
Dim yourValue As String = ""
Using oXmlReader As New Xml.XmlTextReader(sFileName)
While oXmlReader.Read
If (oXmlReader.NodeType = Xml.XmlNodeType.Element) Then
If oXmlReader.Name = QueryNode Then
If oXmlReader.MoveToAttribute(QueryAttribute) Then
yourValue = oXmlReader.Value.
End If
End If
End If
End While
End Using
End Sub
If you have got questions about this, just ask.
Mark the thread as answered if the answer helps you. This helps others who have the same problem ! - Marked As Answer byLing WangMSFT, ModeratorThursday, August 13, 2009 3:19 AM
-
|
| Heslacher Thursday, August 06, 2009 3:37 PM |
You might consider passing the XML to a SQL Stored Procedure(SQLXML) and storing the data in a SQL Server database table. Using SQLXML you can inserrt multiple records with a single statement. That way you could use the power of the database engine to do your processing. I don't remember the limit on XML files size when passed as a parameter but I think its quite large.
Regards.
Mike - Marked As Answer byLing WangMSFT, ModeratorThursday, August 13, 2009 3:19 AM
-
|
| mjhillman Sunday, August 09, 2009 10:29 AM |
You description is a little confusing. So are you using an ArrayList or a DataSet? Can you give more details about what you doing with that xml file data? Does each xml node represent a unique set of input or do the nodes rely on each other?
Is your CPU maxed out? Do you have more multiple cores where you could potentially multi-thread some of it?
Jerry Schulist
Please remember to mark replies which answer your question as answers and vote for replies which are helpful. |
| inetscan Thursday, August 06, 2009 2:29 PM |
I'm reading a xml file into a Dataset > loop through the Dataset with a foreach statement and collect Data from all nodes to save them as Objects into an arraylist. The xml nodes themselve don't rely on each other. The CPU isn't maxed out and the operation is done in a seperate thread.
foreach (DataRow dsXMLRow in dsXML.Tables["NodeNameIchoose"].Rows)
{
collect Data on specific entries
}
Hope this clears things up :)
- Edited byInTo$h Thursday, August 06, 2009 3:12 PM
- Edited byInTo$h Thursday, August 06, 2009 3:13 PM
- Edited byInTo$h Thursday, August 06, 2009 3:13 PM
-
|
| InTo$h Thursday, August 06, 2009 3:11 PM |
I would look into the parallel extensions framework and specifically the Parallel.ForEach method: http://www.microsoft.com/downloads/details.aspx?FamilyId=348F73FD-593D-4B3C-B055-694C50D2B0F3&displaylang=en Simplifies multithreading quite a bit. And check this out: http://blogs.msdn.com/pfxteam/
Jerry Schulist
Please remember to mark replies which answer your question as answers and vote for replies which are helpful. - Marked As Answer byLing WangMSFT, ModeratorThursday, August 13, 2009 3:19 AM
-
|
| inetscan Thursday, August 06, 2009 3:19 PM |
Hi, i would take a look at the XMLReader instead of using a dataset. You can do it like this:
Public Sub XMLReader(ByVal sFileName As String, ByVal QueryNode As String, ByVal QueryAttribute As String)
Dim yourValue As String = ""
Using oXmlReader As New Xml.XmlTextReader(sFileName)
While oXmlReader.Read
If (oXmlReader.NodeType = Xml.XmlNodeType.Element) Then
If oXmlReader.Name = QueryNode Then
If oXmlReader.MoveToAttribute(QueryAttribute) Then
yourValue = oXmlReader.Value.
End If
End If
End If
End While
End Using
End Sub
If you have got questions about this, just ask.
Mark the thread as answered if the answer helps you. This helps others who have the same problem ! - Marked As Answer byLing WangMSFT, ModeratorThursday, August 13, 2009 3:19 AM
-
|
| Heslacher Thursday, August 06, 2009 3:37 PM |
Thanks to both of you!
I'll look into these solutions first thing Monday morning :)
Have a nice weekend!
Cheers,
InTo$h |
| InTo$h Friday, August 07, 2009 2:44 PM |
You might consider passing the XML to a SQL Stored Procedure(SQLXML) and storing the data in a SQL Server database table. Using SQLXML you can inserrt multiple records with a single statement. That way you could use the power of the database engine to do your processing. I don't remember the limit on XML files size when passed as a parameter but I think its quite large.
Regards.
Mike - Marked As Answer byLing WangMSFT, ModeratorThursday, August 13, 2009 3:19 AM
-
|
| mjhillman Sunday, August 09, 2009 10:29 AM |
Mike...I think you just made my day! I never thought of that...for my testenvironment this would be pretty easy to realize...but the the client machine I'm developing for, is pretty darn restricted...but I'll try anyways!
Thanks allot!!!
Cheers,
InTo$h
|
| InTo$h Tuesday, August 11, 2009 1:42 PM |
index > Windows Forms Data Controls and Databinding > Optimizing my Dataset performance