Welcome Guest ( Log In | Register )


Bump Topic Add Reply RSS Feed New Topic New Poll

Outline · [ Standard ] · Linear+

> Is there a way to loop through millions of strings

views
     
TSBetfairTrader
post Mar 24 2019, 11:43 PM, updated 2 months ago

Getting Started
**
Group: Junior Member
Posts: 115

Joined: Jan 2016
I am using javascript. I am given an array that has 1-4 million strings. Is there a secret to loop through them quickly? i just tried the conventional for loop and it took 30 min . sweat.gif
MaGNiFiCo
post Mar 24 2019, 11:57 PM

weeeeeeeeeeeee
******
Group: Senior Member
Posts: 1,583

Joined: Oct 2005
From: Penang



You just looping to print the line or with complex rules?

Can you share the code? Skip the array

https://pastebin.com/
WongGei
post Mar 25 2019, 10:01 AM

Enthusiast
*****
Group: Senior Member
Posts: 862

Joined: Dec 2007
From: Kuala Lumpur
"loop through them" to do search for convert to other string?
Different needs need different algorithm.

OK, I see in your code that you need to convert value of each array element.
Means you have 1~4 millions record to process. There is not shortcut to do that except using faster machine and do parallel processing.

If you are using Javascript, I assume you are using node.js.
Look at this post to enable multi-threading
https://itnext.io/multi-threading-and-multi...js-ffa5bb5cde98

This post has been edited by WongGei: Mar 25 2019, 11:51 AM
narf03
post Mar 25 2019, 04:49 PM

Look at all my stars!!
*******
Group: Senior Member
Posts: 3,880

Joined: Dec 2004
From: Metro Prima, Kuala Lumpur, Malaysia, Earth, Sol


depends on the situation, see if you can create a rule for your data to be sorted, and limit the amount of data to be search or process each time, like how database indexing work, there are billions of record in database, but they can be processed fast, cause of they wont need to loop through all of them.


chibiheroes
post Mar 26 2019, 02:13 PM

Getting Started
**
Group: Junior Member
Posts: 51

Joined: Nov 2009


use web workers for javascript pseudo-multithreaded processing.
dstl1128
post Mar 29 2019, 12:46 PM

Look at all my stars!!
*******
Group: Senior Member
Posts: 4,421

Joined: Jan 2003
QUOTE(BetfairTrader @ Mar 24 2019, 11:43 PM)
I am using javascript. I am given an array that has 1-4 million strings. Is there a secret to loop through them quickly? i just tried the conventional for loop and it took 30 min .  sweat.gif
*
Could you just share what you want to do with each string? Don't tell me you are looping each string and using regex find/replace for every one of them?

If the operations you need to do is relatively deterministic and narrow/small, it might be possible to make it faster. And how varies those each string length were?




TSBetfairTrader
post Mar 29 2019, 10:14 PM

Getting Started
**
Group: Junior Member
Posts: 115

Joined: Jan 2016
QUOTE(dstl1128 @ Mar 29 2019, 12:46 PM)
Could you just share what you want to do with each string? Don't tell me you are looping each string and using regex find/replace for every one of them?

If the operations you need to do is relatively deterministic and narrow/small, it might be possible to make it faster.  And how varies those each string length were?
*
no need already

i actually discovered that looping through 4 million strings is actually instant. its just that i loop it through and ask it to do console.log for everyone of them. that really slows it down.

so as long as i dont console.log every iteration its really fast.


turnmoil
post Apr 3 2019, 10:44 AM

Casual
***
Group: Junior Member
Posts: 363

Joined: Jan 2003
From: The Miau Miaw Niau Miow Land


there a risk doing like that, that may consume processing time later on





Bump Topic Add ReplyOptions New Topic
 

Switch to:
| Lo-Fi Version
0.0602sec    0.87    5 queries    GZIP Disabled
Time is now: 21st May 2019 - 07:00 AM