Welcome Guest ( Log In | Register )

Bump Topic Topic Closed RSS Feed

Outline · [ Standard ] · Linear+

 .net app getting sql data slow from network

views
     
TSnarf03
post Nov 14 2023, 09:10 PM, updated 3y ago

Look at all my stars!!
*******
Senior Member
4,547 posts

Joined: Dec 2004
From: Metro Prima, Kuala Lumpur, Malaysia, Earth, Sol


I have a .net app that connect to SQL server machine using gigabit lan, both pc running on the same gb lan switich, tested no problem copying huge file. almost max out gb connection.

by slow mean, it connect to sql server get like 10k rows of data consists of ~30 columns, and put up to datagridview, takes about 10 seconds to complete.

if i run the app directly in the sql server machine, the exact same operation take less than 1 second.

both pc are i7 or better, no 32G ram, nothing in the background hogging any resource.

I wonder if there are anything limiting the speed of data transfer or displaying data i need to be aware of ?
soul78
post Nov 14 2023, 09:20 PM

Enthusiast
*****
Junior Member
943 posts

Joined: Jul 2005


if both pc got dual network, maybe can try connecting the lan directly bypassing the switch...?
TSnarf03
post Nov 14 2023, 09:27 PM

Look at all my stars!!
*******
Senior Member
4,547 posts

Joined: Dec 2004
From: Metro Prima, Kuala Lumpur, Malaysia, Earth, Sol


QUOTE(soul78 @ Nov 14 2023, 09:20 PM)
if both pc got dual network, maybe can try connecting the lan directly bypassing the switch...?
*
hmm, can try that, but will need a cross cable and fixed ip.

testing with just ssms, select all from a table that consists or 1.3 mil rows, only take 20 seconds in the machine with sql server running, while from the network, 10 minutes also cannot complete, while the load on network is like only 1%(10mbps)
flashang
post Nov 15 2023, 12:24 AM

Casual
***
Junior Member
355 posts

Joined: Aug 2021


you may need to do some profiling speed test.
1. connect to sql server (how many ms)
2. original query and different select or just blank result. (how many ms)
3. put to datagridview.
4. send to user.

5. try different sql client / tool / connect method to sql server.

To figure out where is the issue.


smile.gif



This post has been edited by flashang: Nov 15 2023, 12:28 AM
TSnarf03
post Nov 15 2023, 12:44 AM

Look at all my stars!!
*******
Senior Member
4,547 posts

Joined: Dec 2004
From: Metro Prima, Kuala Lumpur, Malaysia, Earth, Sol


figured out 90%, most likely too many layers of switch and maybe some of them have heavy traffic, if just 1 gigabit switch without any other traffic, able to achieve 100mb/s, that alone speed things up by 10x.

wonder if its possible for the sql server to max out gigabit switch capacity ? like go 9x more faster, achieving 900mb/s
angch
post Nov 15 2023, 09:34 AM

On my way
****
Junior Member
636 posts

Joined: Jul 2006
QUOTE(narf03 @ Nov 14 2023, 09:10 PM)
I have a .net app that connect to SQL server machine using gigabit lan, both pc running on the same gb lan switich, tested no problem copying huge file. almost max out gb connection.

by slow mean, it connect to sql server get like 10k rows of data consists of ~30 columns, and put up to datagridview, takes about 10 seconds to complete.

if i run the app directly in the sql server machine, the exact same operation take less than 1 second.

both pc are i7 or better, no 32G ram, nothing in the background hogging any resource.

I wonder if there are anything limiting the speed of data transfer or displaying data i need to be aware of ?
*
Gut feeling is that you're grabbing the rows one by one, necessitating a network round trip per row. Fetch all of them in one shot, not row by row.

Don't know .net or SQL Server enough, but for Go and PostgreSQL, there's this concept of "single round trip query mode": https://pkg.go.dev/github.com/jackc/pgx/v5#QueryExecMode
to slurp all results at once.

A little more of a leap, but understanding this: https://colin-scott.github.io/personal_webs...ve_latency.html and the implications allows you to visualize performance bottlenecks better.

e.g. in your case, doing best case, one network round trip might be about 0.5ms. 10k rows = 5 seconds already spent sending and waiting for network, at least. It's not about bandwidth. It's about latency.

This post has been edited by angch: Nov 15 2023, 09:39 AM
TSnarf03
post Nov 15 2023, 03:18 PM

Look at all my stars!!
*******
Senior Member
4,547 posts

Joined: Dec 2004
From: Metro Prima, Kuala Lumpur, Malaysia, Earth, Sol


QUOTE(angch @ Nov 15 2023, 09:34 AM)
Gut feeling is that you're grabbing the rows one by one, necessitating a network round trip per row. Fetch all of them in one shot, not row by row.

Don't know .net or SQL Server enough, but for Go and PostgreSQL, there's this concept of "single round trip query mode": https://pkg.go.dev/github.com/jackc/pgx/v5#QueryExecMode
to slurp all results at once.

A little more of a leap, but understanding this: https://colin-scott.github.io/personal_webs...ve_latency.html and the implications allows you to visualize performance bottlenecks better.

e.g. in your case, doing best case, one network round trip might be about 0.5ms. 10k rows = 5 seconds already spent sending and waiting for network, at least. It's not about bandwidth. It's about latency.
*
not by row, not using SP, doing something like select * from table1, no filter


TSnarf03
post Nov 15 2023, 03:20 PM

Look at all my stars!!
*******
Senior Member
4,547 posts

Joined: Dec 2004
From: Metro Prima, Kuala Lumpur, Malaysia, Earth, Sol


I wonder does it matter if i use computer name / ip v6 / ip v4 ?

or it only matters when connection is established, after that it no longer matter(as long as each query doesnt disconnect)
angch
post Nov 15 2023, 11:38 PM

On my way
****
Junior Member
636 posts

Joined: Jul 2006
QUOTE(narf03 @ Nov 15 2023, 03:18 PM)
not by row, not using SP, doing something like select * from table1, no filter
*
Show me your loop.

Are you slurping all data into memory before iterating through the result? Or fetching per row?

ie, are you: (php code, adapt to your current language)

CODE

   $sql = 'select * from table1';
   $stmt = $dbh->prepare($sql, array(PDO::ATTR_CURSOR => PDO::CURSOR_SCROLL));
   $stmt->execute();

   // Some drivers don't batch and prefetch rows, so you can have early termination.
   // This incurs latency, though, but you don't use as much memory.
   while ($row = $stmt->fetch(PDO::FETCH_NUM, PDO::FETCH_ORI_NEXT)) {
       $data = $row[0] . "\t" . $row[1] . "\t" . $row[2] . "\n";
       print $data;
   }


or

CODE

   $sql = 'select * from table1';
   $stmt = $dbh->prepare($sql, array(PDO::ATTR_CURSOR => PDO::CURSOR_SCROLL));
   $stmt->execute();
   $rows = $sth->fetchAll(); // slurp all, less latency incurred, at the cost of memory
   foreach ($row as $row) {
       $data = $row[0] . "\t" . $row[1] . "\t" . $row[2] . "\n";
       print $data;
   }


angch
post Nov 15 2023, 11:38 PM

On my way
****
Junior Member
636 posts

Joined: Jul 2006
QUOTE(narf03 @ Nov 15 2023, 03:20 PM)
I wonder does it matter if i use computer name / ip v6 / ip v4 ?

or it only matters when connection is established, after that it no longer matter(as long as each query doesnt disconnect)
*
No, it doesn't matter.
TSnarf03
post Nov 16 2023, 12:01 AM

Look at all my stars!!
*******
Senior Member
4,547 posts

Joined: Dec 2004
From: Metro Prima, Kuala Lumpur, Malaysia, Earth, Sol


QUOTE(angch @ Nov 15 2023, 11:38 PM)
Show me your loop.

Are you slurping all data into memory before iterating through the result? Or fetching per row?

ie, are you: (php code, adapt to your current language)

CODE

   $sql = 'select * from table1';
   $stmt = $dbh->prepare($sql, array(PDO::ATTR_CURSOR => PDO::CURSOR_SCROLL));
   $stmt->execute();

   // Some drivers don't batch and prefetch rows, so you can have early termination.
   // This incurs latency, though, but you don't use as much memory.
   while ($row = $stmt->fetch(PDO::FETCH_NUM, PDO::FETCH_ORI_NEXT)) {
       $data = $row[0] . "\t" . $row[1] . "\t" . $row[2] . "\n";
       print $data;
   }


or

CODE

   $sql = 'select * from table1';
   $stmt = $dbh->prepare($sql, array(PDO::ATTR_CURSOR => PDO::CURSOR_SCROLL));
   $stmt->execute();
   $rows = $sth->fetchAll(); // slurp all, less latency incurred, at the cost of memory
   foreach ($row as $row) {
       $data = $row[0] . "\t" . $row[1] . "\t" . $row[2] . "\n";
       print $data;
   }

*
no loop, something like this

CODE
private static DataSet SelectRows(DataSet dataset,
   string connectionString,string queryString)
{
   using (SqlConnection connection =
       new SqlConnection(connectionString))
   {
       SqlDataAdapter adapter = new SqlDataAdapter();
       adapter.SelectCommand = new SqlCommand(
           queryString, connection);
       adapter.Fill(dataset); // <----- huge delay here.
       return dataset;
   }
}

angch
post Nov 16 2023, 01:22 AM

On my way
****
Junior Member
636 posts

Joined: Jul 2006
QUOTE(narf03 @ Nov 16 2023, 12:01 AM)
no loop, something like this

CODE
private static DataSet SelectRows(DataSet dataset,
   string connectionString,string queryString)
{
   using (SqlConnection connection =
       new SqlConnection(connectionString))
   {
       SqlDataAdapter adapter = new SqlDataAdapter();
       adapter.SelectCommand = new SqlCommand(
           queryString, connection);
       adapter.Fill(dataset); // <----- huge delay here.
       return dataset;
   }
}

*
Hmm, ok, got it. Low level wireshark or tcpdump might help.

On the other hand, your use of DataAdapter might be problematic: https://learn.microsoft.com/en-us/dotnet/fr...m-a-dataadapter

See the note, copy pasted here:

QUOTE
Using the DataAdapter to retrieve all of a table takes time, especially if there are many rows in the table. This is because accessing the database, locating and processing the data, and then transferring the data to the client is time-consuming. Pulling all of the table to the client also locks all of the rows on the server. To improve performance, you can use the WHERE clause to greatly reduce the number of rows returned to the client. You can also reduce the amount of data returned to the client by only explicitly listing required columns in the SELECT statement. Another good workaround is to retrieve the rows in batches (such as several hundred rows at a time) and only retrieve the next batch when the client is finished with the current batch.
It appears that it not only selects the data, but prepares them individually for updates, locking up the table in the db. Don't know what it does, but Fill() in itself might be the cause of the latency.

Might well be a specific .NET and SQL Server thing, which is out of my depth. Good luck.

This post has been edited by angch: Nov 16 2023, 01:24 AM
3rdEdition
post Nov 16 2023, 01:29 AM

Getting Started
**
Junior Member
213 posts

Joined: Sep 2012

Run Wireshark on relevant interface and execute same query.
Compare both pcaps and find out the difference.

SUSifourtos
post Nov 16 2023, 01:41 AM

Look at all my stars!!
*******
Senior Member
2,256 posts

Joined: Feb 2012



how you do the Query???
how is this related to Network bandwidth....

10mb of data already astronomical data...
even the worst network setup today can handle without sweating.


The way you query is the problem,
Dothan
post Nov 16 2023, 07:54 AM

Dingle Berries
******
Senior Member
1,054 posts

Joined: Jan 2003


without much information of what technology stack, we cannot pinpoint where is the bottleneck.
praetorion
post Nov 16 2023, 10:20 AM

Getting Started
**
Junior Member
289 posts

Joined: Jan 2003
From: MAS
The datagridview is taking that long to bind and display the 10k records. nothing to do with the network.
silverhawk
post Nov 17 2023, 12:39 AM

Eyes on Target
Group Icon
Elite
4,956 posts

Joined: Jan 2003


I doubt network is the real issue

10k rows is not a lot, even to transfer over the network as a single payload. So your problem is definitely not in the network, but somewhere in your code.

Must be something in your code that's creating latency, which is not obvious when run locally, but easily adds up over the network.

Lets say you are fetching 10k records, and some loop or something you're doing is introducing a network latency (e.g. server callback, logging, etc). An average of 1 extra ms for each row is 10,000ms which is 10 seconds.
nyem
post Nov 17 2023, 01:49 AM

Enthusiast
*****
Senior Member
751 posts

Joined: Jan 2007


I'd do a test from cmd line to see if the lag also happen when not using your code

CODE
sqlcmd -Q "SELECT * FROM table" -o MyOutput.txt




Topic ClosedOptions
 

Change to:
| Lo-Fi Version
0.0202sec    0.56    5 queries    GZIP Disabled
Time is now: 23rd December 2025 - 04:41 AM