Can windows XP pro be made to use more RAM when copying large files? - Techist - Tech Forum

Go Back   Techist - Tech Forum > Computer Software > Microsoft Windows and Software
Click Here to Login
Closed Thread
 
Thread Tools Display Modes
 
Old 07-11-2005, 06:42 AM   #1 (permalink)
Newb Techie
 
Join Date: Jul 2004
Posts: 11
Default Can windows XP pro be made to use more RAM when copying large files?

From what I`ve seen win only uses small chunks of RAM even when copying massive files. It causes the HDD heads to thrash and really slows the copy process down while hundreds of MB of RAM sit there unused!

Or is win taskman not showing the true RAM usage? I.E. win is using the 100's MB RAM but it's still showing it as free incase some app needs it....
__________________

Maestro is offline  
Old 07-11-2005, 11:18 AM   #2 (permalink)
Super Techie
 
Join Date: May 2005
Posts: 479
Default

well, apparently windows split all RAM usage with the page file.. so it works a bit different from the *nix.. as *nix utilizes everything in RAM before move on to swap file, but windows likes to keep RAM free (it can cause speed problem when your running huge app, like photoshop with 50mb file).. i don't know yet of any way that can change this, but you can try googling
__________________

__________________
lisp hacker :D
running: FreeBSD 5.4 - still learning :D
develop with: SBCL + emacs for lisp, Anjuta IDE +gcc for c, SPE for python..
browse with: opera
furtivefelon is offline  
Old 07-11-2005, 05:24 PM   #3 (permalink)
Newb Techie
 
Join Date: Jul 2004
Posts: 11
Default

First thing I tried was the Google. Google is a friend!

Sadly I haven't been able to find anything that answers this query. I did find lots of stuff about 3rd party programs to do file copying faster than win but I'd rather fix windows shortcomings than keep installing more stuff and using a separate app isn't going to be as intuitive as drag & drop in win.

I dunno, maybe windows is using the RAM and taskman just doesn't show it? I do recall reading some thread on a forum some time back that win only copies files in small chunks. I hope that's wrong but it seems like it's correct info so far. Just bugs me that when copying large files all that RAM is just sitting there going to waste, doing nothing, when it could be buffering a big read/write chunk. Seems to waste the drive performance to me. Even the OS on a crusty old Amiga would use large RAM chunks to speed file copying. Does it leave huge amounts of unused RAM when copying large files on your machine?

I think I'll try and come up with some more keywords for google in the meantime.
Maestro is offline  
Old 07-12-2005, 12:25 AM   #4 (permalink)
Super Techie
 
Join Date: May 2005
Posts: 479
Default

mmm.. i'm not really sure using more RAM would be much help.. since your harddrive only has certain speed (I/O (Input/Output)), and even if you commit huge chunk to memory, you still is restricted by your harddrive I/O speed (normally memory works multiple times faster than harddrive can ever hope to achieve).. and i recall i read somewhere that *most* hard drives handle small chunks effeciently, so i'm guessing that smaller chunk at a time would improve performance? i might be totally wrong on this, so please correct me if i am

oh yah, another guess would be that windows tries to be a multitask OS, so it tries to limit each program's resource allocation so to allow other program work faster??
__________________
lisp hacker :D
running: FreeBSD 5.4 - still learning :D
develop with: SBCL + emacs for lisp, Anjuta IDE +gcc for c, SPE for python..
browse with: opera
furtivefelon is offline  
Old 07-12-2005, 11:44 AM   #5 (permalink)
Newb Techie
 
Join Date: Jul 2004
Posts: 11
Default

The problem isn't when copying between 2 drives (unless other processes are using either source/destination/both HDD during a copy process), it's actually only when copying from & to the same drive where it always takes at least twice as long to copy a given file than it would when using a separate source drive and destination drive (if both drives are similar performance/spec).

EG. In my case it takes 11sec to copy a 529MB file from my 80GB maxtor to my 120GB maxtor (both ATA133 7200rpm 2MB cache) whereas duplicating the same file on the 120GB maxtor results in a surprising 33secs. I was expecting more like 25secs tops. This suggests to me that windows is copying files in small chunks and causing more head seeking, ergo wasting time and drive performance. This could be avoided if much larger chunks were read/written in this situation.

If you have 2 similar drives on same system, what kind of time ratio do you get when doing drive 2 drive file copy and same drive file duplication?
Maestro is offline  
Old 07-12-2005, 11:29 PM   #6 (permalink)
Super Techie
 
Join Date: May 2005
Posts: 479
Default

mmm.. depends.. you see, if you have one huge partition of 120 gig of harddrive, you shouldn't have any delay in copying from one folder to the next (as i believe they just change the addressing.. specifics, i don't know..)

if you have multiple partition on one hard drive, it's reasonable to copy from one partition to another takes longer than copy between drives.. it has nothing much to do with the I/O speed here (as we are talking about the same drive).. when you copy from one drive to the next, you have two drives spinning at the same speed, so it copies at the full 7200 rpm for data.. and if you copy between partitions, you are spliting that 7200 rpm into chunks so that you can move stuff from one place in the same drive to the next.. so it reasonably takes longer than the two drive.. how long exactly, i'm not sure, as i have never tried, whenever i start copying something, i just leave it and go to washroom/get a drink or something. it's usually fast (< 1 min) that i can go back and be done..

i think 600mb stuff in 32 sec is not that long? hehe

oh yah, if you don't understand anything (or i just write it in a confusing way, or you don't agree), please do ask/challenge
__________________
lisp hacker :D
running: FreeBSD 5.4 - still learning :D
develop with: SBCL + emacs for lisp, Anjuta IDE +gcc for c, SPE for python..
browse with: opera
furtivefelon is offline  
Old 07-13-2005, 08:32 AM   #7 (permalink)
Newb Techie
 
Join Date: Jul 2004
Posts: 11
Default

Actually you will get a delay regardless of partition size as it doesn't make any difference whether it's copying on same partition, between partitions on same drive, or between partitions on different drives, it is still doing a file copy (you end up with 2 copies of same file). What you are referring to is a file move within same partition, where the file data isn't copied, just the directory structure is updated to reflect the new location of the file. However, if a file is moved between partitions on same or different drives, then the file data has to be copied and the source then deleted. So in that case a file move is the same as doing a file copy followed by a source file delete.

It actually has everything to do with I/O because the maximum data transfer rate is related to the rotational speed of the drives involved in the copying process. Regardless of whether the data is being copied on same drive or different drives, it still has to travel across the ATA bus to the computers memory and back again. As I said before, when duplicating files on same drive, the drive head(s) must seek back & forth to the source location (for reads) and the destination location (for writes) therefore involving much more drive head seeking. This is where buffering more data will help as it is increasing the data chunk size and this reduces the amount of head seeks necessary to complete the copy process.

No it isn't a long time if you do something else while copying but if you scale the filesize up, the amount of wasted time induced by head seeks caused by copying small chunks, increases and can amount to quite a lot of time. This is annoying if you need to get tasks done involving the copied file and must wait for it each time. If you need to do this many times a day the time wasted amounts to quite a lot with very big (xxGB sized) files. The only solution to this I have so far is to use 2 drives but sometimes I need to duplicate on same drive.

As I said, copying between 2 similar performing drives versus copying on either one of those drives only, is usually about a minimum of 1:2 ratio because one drive performs reads while the other writes, whereas a single drive has to do both reads and writes which it can't perform simultaneously, it has to do one or the other to achieve the task. What I don't like about windows is that it only manages about 1:3 ratio which I think is excessive and due to it copying in small chunks resulting in an increase of unecessary head seeking. This is why I would like to tweak windows to use more RAM during large file copies on same drive. It can also be helpful when copying between 2 drives where 1 or both are in use by other processes during a copy and/or 1 drive has much greater performance than the other.
Maestro is offline  
Old 07-13-2005, 10:15 PM   #8 (permalink)
Super Techie
 
Join Date: May 2005
Posts: 479
Default

mmm.. i thought I/O meant the bandwidth of the transfer.. guess i was wrong on that one :P

could you turn off the pagefile in windows? if you can, try turning it off to see if it will use hte RAM more or less (just a guess )

other than that, i don't have much clue about hardware except the basic ones... i'll try to do some more research on ram/pagefile and get back to you as soon as i find something
__________________
lisp hacker :D
running: FreeBSD 5.4 - still learning :D
develop with: SBCL + emacs for lisp, Anjuta IDE +gcc for c, SPE for python..
browse with: opera
furtivefelon is offline  
Old 07-14-2005, 06:51 AM   #9 (permalink)
Ultra Techie
 
Join Date: Jul 2005
Posts: 530
Send a message via AIM to TheHeadFL
Default

You seem to be assuming that buffering larger amounts of data would neccesarily speed up your copy on the same drive.

This assumption does not hold for a few reasons....

First, there is no guarantee as to how the underlying data is stored on the physical media. Remember you are copying logical blocks of data, and you don't know anything about the contents of the underlying locations. There is no guarantee that the file data is stored in contigious space in the first place, so you know nothing about the seek time between each successive request being fulfilled. There is just no way to say definitively that you are actually seeking less.

Secondly, the controllers of a drive cannot handle data instantaneously. Data isn't copied into the drive buffers at the speed the disk actually can physically write it, since this would be a monumental waste of processor time. Data is already passed in discrete chunks to maximize transfer rate and minimize the number of cycles your process is waiting on the I/O bus. Data is going to be cached up and then written to the disk. While the drives own controller is actually handling the mechanics of the write, your CPU is already context switched out to some other task. When it is ready again, the same thing happens. This is important because your process can send data over the I/O bus to the target drive (if it was physically a different drive) *at the same time* as the source drive was actually fetching and caching the requested data. You're getting a two for one situation here. This benefit is obviously null with the same drive.

Third, obviously, is specific to the concept of on-drive cache memory. The reason drives have cache is so that frequently accessed chunks of memory do not have to be fetched physically from the media each time they are operated on. Copying a very large amount to physical memory may seem like a good idea, but you're effectively making your cache useless. If you are reading in 200mb of contigious data from your logical file, there is precisely ZERO chance that you are going to have that data available in cache (either on CPU *or* on-disk) when it is to be written, since you clearly do not have nearly that kind of cache memory. Instead, it has to be subsequently fetched from physical memory (slow) and then sent back over the I/O bus (slower) and the thread waits while the I/O operation completes and the disk becomes available for writing again (worst of all) before it can continue.

Remember, there isn't only one processor in your computer. You have lots of specialized processors. There is a reason large drive caches and intelligent drive electronics make disk access faster.
__________________
Desktop machine: 2 x Opteron 246, Asus K8N-DL, 2GB PC3200 ECC Reg., XFX GeForce 6600GT, 74gb WD Raptor, 2 x 19\" LCDs, Windows XP x64
Server machine: Intel P4 3.0GHz 2MB EM64T, ECS i865pe, 1GB PC3200, 36gb WD Raptor, Windows Server 2003
Laptop: Dell Inspiron 9100 (Intel P4 3.2GHz 1MB Prescott, i865pe, 512MB PC3200, Mobility Radeon 9700, DVD+R/DL Burner), Windows XP
Linux: P3 450Mhz, 386MB ram, Slackware 10.1 (Running mySQL/Apache)
TheHeadFL is offline  
Old 07-20-2005, 06:59 PM   #10 (permalink)
Newb Techie
 
Join Date: Jul 2004
Posts: 11
Default

I mentioned the points you make in your first 2 paragraphs regarding file fragmentation and dual drives vs single. As I said, there is no guarantee that using large mem chunks will decrease file copy time if the source file is fragmented badly. Fair enough that it could be the case that apart from initial read access, the drive uses read ahead caching and subsequent read accesses are immediately served by the cache, possibly giving the drive enough time to seek to the read position again in time for the cached read data being exhausted. I honestly don't know there for sure. I just wanted to see if I could force windows to give the large mem chunk a shot to see if it gave any improvements during single drive file duplications, or not, as the case may be.


How would you explain the increase in file copy times that I am getting on same drive if not read/write seek related? I think 3 fold is excessive to be honest. 2 fold yes, but not 3. During my tests I checked that the source file was not fragmented. Same for destination free space.


How do your drives fair in a drive to drive file copy vs same drive file duplication?
__________________

Maestro is offline  
Closed Thread

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off




Copyright 2002- Social Knowledge, LLC All Rights Reserved.

All times are GMT -5. The time now is 04:26 AM.


Powered by vBulletin® Version 3.8.8 Beta 1
Copyright ©2000 - 2017, vBulletin Solutions, Inc.