Hi, Now that we're facing Direct3D 9 feature completion soon - VTF support coming in from Henri, High order patches and thread safety from me - the d3d work will change a bit soon. Instead of adding new features we'll debug games and improve performance. It would be good if we had some automated testing framework to support the process :-)
Unfortunately this isn't easy. Games do not render exactly, so verifying that a game doesn't show any visual regressions automatically may be more effort than reward. Here I think we'll have to rely on users testing the games regularly and testing regressions.
I am more concerned about the performance thing. It is very easy to make some performance improvement that does more harm than good. Also benchmarks take quite some time and are affected by many external parameters. So I'd like some automated and controlled performance testing environment. Instead of building one from scratch it would be cool if we could reuse existing software.
cxtest comes to my mind(www.cxtest.org). It is a test suit built by codeweavers for running regression tests on Windows apps. It can remote control applications using keyboard input and compares screenshots to reference ones. It could be used to start benchmarks automatically. A plus here is that some machines run nightly tests. However, those are VMs, so no 3D acceleration. A problem is also reading back the results. Does anyone know some applications for doing that? Windows or Linux apps, commercial or open source ones(prefered, though).
Then we need benchmarking apps. Here regular Windows benchmarks come to my mind(3DMarkXYZW), games with benchmark facilities(hl2, ut, ...). We can record tests, get a reference performance counter from windows and run nightly tests(or manually started tests). A problem we're facing is reading back the results. Taking screenshots and running OCR on them is error prone, we need a way to get the numbers from the benchmarks, not an image of the numbers.
Now we need a concrete collection of benchmarks. A mixture of popular Benchmarks like 3DMark would be good, and some more cientific benchmarks. So far I have:
3DMark: Popular benchmark for testing the overall performance, doesn't give very detailed results though. Controlling: Standard windows controls, keyboard events should do the job Result readback: Can write results to a file, but must take care about the licensing. Not everything is free, most likely a license is needed.
Half-Life 2: Nice facility of recordable timedemos. Can record average game demos, but also demos which stress specific features. Controlling: Command line parameters Result readback: Can be asked to copy the console into the clipboard
UT2004: Popular engine. Has a Linux port, but many mods only ship the d3d renderer. Controlling: Unknown Result readback: unknown.
DirectX sdk demos, nvidia / ati demos: Stress specific rendering features. Good for testing performance(with vsync removed), or a bad idea? Controlling, Readback: Source code is available and can be modified. License, especially possible benchmark clauses?
Other suggestions?
Another nice-to-have thing would be a game independent profiling method, simmilar to the +fps debug channel, just more detailed. Some games do not have built-in benchmarking, it would still be nice to use them. That means facing some new problems? How to tell loading from in-game rendering? How to remove control those games in a repeatable way, especially if there are some random elements in the game? More detailed data from wined3d could also be used to draw nice graphs for other benchmarks :-)
We'll need some hardware diversity too. It would be nice if the tests worked on Linux and MacOS, and a variety of hardware was used. I want at least nvidia, ati and intel gpu's, older and newer ones. I could also revive my ATI mach64 :-)
Any other comments or suggestions? I think once I'm done with the current exam season and my patch backlog is in I will retire from direct wined3d development for a few days / weeks to set up a testing environment, then start fixing games.
On 25/06/07, Stefan Dösinger [email protected] wrote:
3DMark: Popular benchmark for testing the overall performance, doesn't give very detailed results though. Controlling: Standard windows controls, keyboard events should do the job Result readback: Can write results to a file, but must take care about the licensing. Not everything is free, most likely a license is needed.
Afaik 3DMark can also generate performance graphs.
Other suggestions?
We could always try writing something ourselves, I guess :-) The problem there is going to be creating something that's representative though.
random elements in the game? More detailed data from wined3d could also be used to draw nice graphs for other benchmarks :-)
NVPerfSDK could potentially help there, if the license permits.
We'll need some hardware diversity too. It would be nice if the tests worked on Linux and MacOS, and a variety of hardware was used. I want at least nvidia, ati and intel gpu's, older and newer ones. I could also revive my ATI mach64 :-)
Any other comments or suggestions? I think once I'm done with the current exam season and my patch backlog is in I will retire from direct wined3d development for a few days / weeks to set up a testing environment, then start fixing games.
One issue I see is actually interpreting the results. When is a performance drop large enough to be a problem? Sometimes a change will slightly reduce performance for some applications, but significantly improve it for others.
Afaik 3DMark can also generate performance graphs.
I prefer to have the raw data myself, to do my own processing, so I'm not sure if that helps us. But graphcs are just a nice to have, we don't need them.
We could always try writing something ourselves, I guess :-) The problem there is going to be creating something that's representative though.
Ya, I don't think our own tests will help us much.
random elements in the game? More detailed data from wined3d could also be used to draw nice graphs for other benchmarks :-)
NVPerfSDK could potentially help there, if the license permits.
I'm concerned about the NV in NVPerfSDK :-) Of course its a tool we can use, but we shouldn't depend on it.
One issue I see is actually interpreting the results. When is a performance drop large enough to be a problem? Sometimes a change will slightly reduce performance for some applications, but significantly improve it for others.
Thats a good point, and I've been thinking about it a bit. All we can use is some heuristics. Throwing a warning on a single 0.01 fps drop is an overkill because there are minor differences between runs, even with exactly the same code. On the other hand, if a number of 0.5 fps drops sum up to a drop from 80 to 75 fps it is something that should perhaps be investigated, so we should check against a fixed reference value rather than the last day's result.
Wether a performance drop is a real regression or an unavoidable side effect from something else, like a rendering correctness fix is something we have to decide per incident. The automated tests can only generate a warning(e.g. per mail). Then we have to look at the warning and decide what to do with it. We could either fix the regression, if it is one, or adjust the reference value.
On 25/06/07, Stefan Dösinger [email protected] wrote:
Afaik 3DMark can also generate performance graphs.
I prefer to have the raw data myself, to do my own processing, so I'm not sure if that helps us. But graphcs are just a nice to have, we don't need them.
I think it has the ability to export frame timings to Excel format. Never used that myself though, and I've got no idea if it even works in Wine.
On Mon, 25 Jun 2007, Stefan Dösinger wrote: [...]
One issue I see is actually interpreting the results. When is a performance drop large enough to be a problem? Sometimes a change will slightly reduce performance for some applications, but significantly improve it for others.
Thats a good point, and I've been thinking about it a bit. All we can use is some heuristics. Throwing a warning on a single 0.01 fps drop is an overkill because there are minor differences between runs, even with exactly the same code. On the other hand, if a number of 0.5 fps drops sum up to a drop from 80 to 75 fps it is something that should perhaps be investigated, so we should check against a fixed reference value rather than the last day's result.
I don't think we can automatically flag performance drops as regressions / errors.
First you would have to compare the performance of the current run against that of one of the previous runs. This runs counter to the norm which is that each run is self contained and completely independent of the other runs.
Second, there is no garantee that all users running the nightly tests will have an otherwise idle machine. For instance I run cxtests every night but on some days my machine may still be finishing a backup, or recording a movie, or compiling wine with gcc 2.95, etc. Each of those could cause significant performance variations that could be mistaken for a performance bug.
Also, if you only compare one run to the next you will only detect a performance drop on one day, which can easily be dismissed as noise due to the above, and everything will be normal from then on. So you have to do a more long term analysis.
So I think a more promising approach is to just collect data and generate graphs. Then we can have human beings look at the graphs and investigate when something looks weird, like if the performance on 90% of the machines drops around the same date.
You could also write some software to automatically analyze this data on the server but that seems overkill.
Hi, With Francois' help I could build a test package for 3DMark2000 now:
http://stud4.tuwien.ac.at/~e0526822/3dmark2000.tar.bz2
It runs two tests, one at 640x480, another one at 1024x768, writes the results to a file and reads the file back. No benchmark results are sent to the server yet(just ok / fail).
Steps to run the test: 1) install cxtest from http://www.cxtest.org/download . I recommend installing it with a seperate user account, since the tests scripts mess with files in $HOME. cxtest will download the wine git tree and compile wine, so you need some free space. cxtest installs some sample test packages too(picasa, winzip, pptviewer).
It was a bit tricky for me to set cxtest up, since it needs Xvnc, has some problems(non-fatal it seems) with git 1.5, and a problem with the nvidia driver interfered(I couldn't run 2 Wine instances from different user accounts, I got a crash in libglcore). When you try to *record*, not play back a test, make sure to have $DISPLAY unset(vnc is used)
2) Download my test package and unpack it in ~/cxtest/
3) Run the test WITHOUT using vnc. Obviously you need $DISPLAY set, and 3D acceleration available:
~/cxtest/cxtest_run/bin/run.sh --run 3dmark2000 --verbose --no-user --no-dependency --wine ~/cxtest/wine/wine --no-vnc
So whats left? *) Make the Email system send custom test return data, not just success / failure *) extend the server side infrastructure to handle that test data. *) Write tests for more games
Am Donnerstag, 28. Juni 2007 18:05 schrieb Stefan Dösinger:
So whats left? *) Make the Email system send custom test return data, not just success / failure
I found a simple solution for that: I just dump the results onto the log channel, which is sent to the server which can then grep the numbers out.
*) extend the server side infrastructure to handle that test data.
Any volunteers? I tried to avoid web development as good as possible and I'm pretty crappy at it :-/
*) Write tests for more games
Will write more :-)
With Francois' help I could build a test package for 3DMark2000 now:
http://stud4.tuwien.ac.at/~e0526822/3dmark2000.tar.bz2
It runs two tests, one at 640x480, another one at 1024x768, writes the results to a file and reads the file back. No benchmark results are sent to the server yet(just ok / fail).
Stefan, this link is not valid anymore, could you make the file available again?
Thanks, Martin
Am Dienstag, 31. Juli 2007 12:28 schrieb martin pilka:
With Francois' help I could build a test package for 3DMark2000 now:
http://stud4.tuwien.ac.at/~e0526822/3dmark2000.tar.bz2
It runs two tests, one at 640x480, another one at 1024x768, writes the results to a file and reads the file back. No benchmark results are sent to the server yet(just ok / fail).
Stefan, this link is not valid anymore, could you make the file available again?
Oops, I deleted the file accidentally when cleaning up old junk on that server. I've reuploaded it.
Hello Stefan,
link works, I was able to run your test. Installation is ok, however test itself ends with '3DMark Error': '3DMark needs DirectX 7 and proper drivers installed in order to run'. This happens without CxTest as well, with latest WineHq. Note that DirectX7 or newer is correctly detected during installation process.
Is it a bug in my configuration, or did I hit some Wine error?
Thanks, Martin
Stefan Dösinger wrote:
Am Dienstag, 31. Juli 2007 12:28 schrieb martin pilka:
With Francois' help I could build a test package for 3DMark2000 now:
http://stud4.tuwien.ac.at/~e0526822/3dmark2000.tar.bz2
It runs two tests, one at 640x480, another one at 1024x768, writes the results to a file and reads the file back. No benchmark results are sent to the server yet(just ok / fail).
Stefan, this link is not valid anymore, could you make the file available again?
Oops, I deleted the file accidentally when cleaning up old junk on that server. I've reuploaded it.
Am Dienstag, 31. Juli 2007 16:42 schrieb martin pilka:
Hello Stefan,
link works, I was able to run your test. Installation is ok, however test itself ends with '3DMark Error': '3DMark needs DirectX 7 and proper drivers installed in order to run'. This happens without CxTest as well, with latest WineHq. Note that DirectX7 or newer is correctly detected during installation process.
Is it a bug in my configuration, or did I hit some Wine error?
Are you running on VNC? Do you have opengl support? This error occurs when I run on a system without opengl support.
Stefan Dösinger wrote:
Am Dienstag, 31. Juli 2007 16:42 schrieb martin pilka:
Hello Stefan,
link works, I was able to run your test. Installation is ok, however test itself ends with '3DMark Error': '3DMark needs DirectX 7 and proper drivers installed in order to run'. This happens without CxTest as well, with latest WineHq. Note that DirectX7 or newer is correctly detected during installation process.
Is it a bug in my configuration, or did I hit some Wine error?
Are you running on VNC? Do you have opengl support? This error occurs when I run on a system without opengl support.
It is without CxTest and without VNC. I do have HW acceleration and OpenGL (GLX) enabled. Attached is output of xdpyinfo.
I am going to try whether 3DMark2001 works.
Martin
Hello Stefan!
It is without CxTest and without VNC. I do have HW acceleration and OpenGL (GLX) enabled. Attached is output of xdpyinfo.
I am going to try whether 3DMark2001 works.
3DMark2001 behaves the same way. Note this is long lasting bug, I remember on WineConf 2006 I was able to run 3DMark2000 under CrossOver, but only after the fresh installation. Subsequent runs failed with the same error we have today (DirectX detection failed). It is probably configuration dependent (~/.wine).
However, I was able to run it on another machine (Mac OS X) using latest CrossOver nightly build. There were some minor issues, but after a while I was able to get benchmark.txt file. I submitted that file to CxTest server several times, to test new Direct3D evaluation functionality. It is here:
http://www.cxtest.org/product-d3d-eval?id_product=3&m_from=7&d_from=...
Please have a look whether this is (roughly) what you need.
To make this useful, we need to solve 'DirectX detection failed' bug, so we can run real tests nightly. Could you send me ~/.wine directory you are using with latest WineHq?
Thanks, Martin
To make this useful, we need to solve 'DirectX detection failed' bug, so we can run real tests nightly. Could you send me ~/.wine directory you are using with latest WineHq?
Do you have HKCU/Software/Wine/Direct3D/DirectDrawRenderer set to "gdi" perhaps? This disables D3D support altogether. You could try with a fresh .wine, it should run 3dmark2000 and 2001 out of the box.
I'm not sure wether I should really attempt to send you my ~/.wine, it is full with games and has a size of 51GB :-) . I could send you the .reg files though.
Do you have HKCU/Software/Wine/Direct3D/DirectDrawRenderer set to "gdi" perhaps? This disables D3D support altogether. You could try with a fresh .wine, it should run 3dmark2000 and 2001 out of the box.
I tried latest WineHq, it created ~/.wine directory and installed 3DMark2000 correctly, however when I try to run I get same complains: "3DMark needs DirectX7 and proper drivers installed". Also, this is printed on console:
cxtest@mpilka:~$ ~/cxtest/wine/wine ./.wine/drive_c/Program\ Files/MadOnion.com/3DMark2000/3DMark2000.exe err:ddraw:DDRAW_Create Couldn't load WineD3D - OpenGL libs not present? err:ddraw:DDRAW_Create Couldn't load WineD3D - OpenGL libs not present? err:ddraw:DDRAW_Create Couldn't load WineD3D - OpenGL libs not present?
Note ~/.wine directory generated by Wine does not mention HKCU/Software/Wine/Direct3D/DirectDrawRenderer key at all. Also, glxgears runs fine, xdpyinfo says GL is there (ATIFGLEXTENSION, ATIFGLRXDRI, GLX, SGI-GLX extensions are present).
Martin
Am Montag, 6. August 2007 16:49 schrieb martin pilka:
Do you have HKCU/Software/Wine/Direct3D/DirectDrawRenderer set to "gdi" perhaps? This disables D3D support altogether. You could try with a fresh .wine, it should run 3dmark2000 and 2001 out of the box.
I tried latest WineHq, it created ~/.wine directory and installed 3DMark2000 correctly, however when I try to run I get same complains: "3DMark needs DirectX7 and proper drivers installed". Also, this is printed on console:
cxtest@mpilka:~$ ~/cxtest/wine/wine ./.wine/drive_c/Program\ Files/MadOnion.com/3DMark2000/3DMark2000.exe err:ddraw:DDRAW_Create Couldn't load WineD3D - OpenGL libs not present? err:ddraw:DDRAW_Create Couldn't load WineD3D - OpenGL libs not present? err:ddraw:DDRAW_Create Couldn't load WineD3D - OpenGL libs not present?
is wined3d built?
Hello Martin
If you cd into the 3DMark2000 install directory and run it from there does it work?
Tom
Thanks Tom, that was it. I got a bit further, the message I got this time is:
3DMark could not initialize your 3D Accelerator. Try to downgrade your display settings.
I played with resolutions and xorg.conf file quite a bit, no joy so far, could you send me your xorg.conf file?
Thanks, Martin
Can you try this, set Wine to run in a virtual desktop of 1024x768 and set X at 16bpp and run 3DMark2000 in win98 mode. then cd into the install directory and see if that works.
Tom
On 8/8/07, martin pilka [email protected] wrote:
Hello Martin
If you cd into the 3DMark2000 install directory and run it from there does it work?
Tom
Thanks Tom, that was it. I got a bit further, the message I got this time is:
3DMark could not initialize your 3D Accelerator. Try to downgrade your display settings.
I played with resolutions and xorg.conf file quite a bit, no joy so far, could you send me your xorg.conf file?
Thanks, Martin
Francois Gouget wrote:
You could also write some software to automatically analyze this data on the server but that seems overkill.
Yes, we tried this while ago and it was really overkill. I believe results should be analyzed by man, but he needs user friendly formatted information for this. We are trying to provide such information i.e. here:
http://www.cxtest.org/product-evaluation?id_product=3&id_failure=170&...
It is clear that there was a regression between 070109 and 070110. Martin
On Mon, Jun 25, 2007 at 03:40:42PM +0200, Stefan Dösinger wrote:
Hi, Now that we're facing Direct3D 9 feature completion soon - VTF support coming in from Henri, High order patches and thread safety from me - the d3d work will change a bit soon. Instead of adding new features we'll debug games and improve performance. It would be good if we had some automated testing framework to support the process :-)
Unfortunately this isn't easy. Games do not render exactly, so verifying that a game doesn't show any visual regressions automatically may be more effort than reward. Here I think we'll have to rely on users testing the games regularly and testing regressions.
cxtest comes to my mind(www.cxtest.org). It is a test suit built by codeweavers for running regression tests on Windows apps. It can remote control applications using keyboard input and compares screenshots to reference ones. It could be used to start benchmarks automatically. A plus here is that some machines run nightly tests. However, those are VMs, so no 3D acceleration. A problem is also reading back the results. Does anyone know some applications for doing that? Windows or Linux apps, commercial or open source ones(prefered, though).
Other suggestions?
cxtest has already impressive infratructure (installation, creation of clean environment) and nice scripting. However, it requires Metacity WM and VNC.
I'm personally using AutoHotkey(http://www.autohotkey.com) as an automatic testing framework backend. It's a Windows application, so it can be run through Wine itself and it's sandboxed from X11.
I can imagine Winebot packages to be easily converted to an automatic testing engine. Winebot actually covers automated installation and creation of clean independent environment.
Running applications with different log levels, tracing output, comparing screenshots with different levels of fuzzy comparaion - it's not faraway.
AFAIK cxtest has the better functionality now.
Regards Vit
cxtest has already impressive infratructure (installation, creation of clean environment) and nice scripting. However, it requires Metacity WM and VNC.
The Metacity WM doesn't really concern me, the VNC thing however does. This makes game testing harder since VNC isn't a part of the average game setup. We don't have to transfer all the rendering screenshots over VNC though. I am CCing Martin Pilka on this, maybe he can give us some more insight.
I'm personally using AutoHotkey(http://www.autohotkey.com) as an automatic testing framework backend. It's a Windows application, so it can be run through Wine itself and it's sandboxed from X11.
It looks interesting, this would also allow us to run automated tests on Windows, perhaps to track the effect of driver updates, etc.
I can imagine Winebot packages to be easily converted to an automatic testing engine. Winebot actually covers automated installation and creation of clean independent environment.
I don't know if we have to automate the installations as well. Reinstalling the apps excludes a few factors and makes the tests more repeatable, on the other hand it is not something that occurs in the average gaming environment. Not reinstalling can bring up problems related to the game's state on the disk, like fragmenting game caches. When we see a problem we can reset the environment manually to exclude the disk content as the cause. Automated game installation is a nice to have, but not a strict requirement IMHO.
Not reinstalling would ease the use of some controller application that runs inside wine, since constant installing and removing of the app can leave garbage in the C drive and the registry over time. On the other hand, an external application could remove the wineprefix and start with a completely clean Windows environment.
Running applications with different log levels, tracing output, comparing screenshots with different levels of fuzzy comparaion
- it's not faraway.
How would fuzzy comparison work?
On Mon, 25 Jun 2007, Stefan Dösinger wrote:
cxtest has already impressive infratructure (installation, creation of clean environment) and nice scripting. However, it requires Metacity WM and VNC.
The Metacity WM doesn't really concern me, the VNC thing however does. This makes game testing harder since VNC isn't a part of the average game setup. We don't have to transfer all the rendering screenshots over VNC though. I am CCing Martin Pilka on this, maybe he can give us some more insight.
cxtest's run.sh script has a --no-vnc option. I know I have once run a test outside of VNC so VNC is not a hard requirement. cxtest will probably complain about it though, and you might lose some functionality like automatic screenshots on failures. The best would be to try it out.
cxtest's run.sh script has a --no-vnc option. I know I have once run a test outside of VNC so VNC is not a hard requirement. cxtest will probably complain about it though, and you might lose some functionality like automatic screenshots on failures. The best would be to try it out.
--no-vnc works fine for me for playing tests, but it does not work for recording tests. Thats pretty bad because the vnc server does not even have opengl, not even a software renderer. Because of that the benchmarks refuse to start up at all. I tried to write the test scripts manually(I don't need many clicks after all), but finding out all the button IDs sounds bad :-/ I'll try my luck with keystrokes now...
On 6/27/07, Stefan Dösinger [email protected] wrote:
--no-vnc works fine for me for playing tests, but it does not work for recording tests. Thats pretty bad because the vnc server does not even have opengl, not even a software renderer. Because of that the benchmarks refuse
Could we add an extension to vnc to use Mesa?
On 28/06/07, Steven Edwards [email protected] wrote:
Could we add an extension to vnc to use Mesa?
I suspect it would be non-trivial, but more importantly I think it would defeat the purpose of measuring performance on different hardware configurations.
Am Donnerstag, 28. Juni 2007 09:09 schrieb H. Verbeet:
On 28/06/07, Steven Edwards [email protected] wrote:
Could we add an extension to vnc to use Mesa?
I suspect it would be non-trivial, but more importantly I think it would defeat the purpose of measuring performance on different hardware configurations.
The problem with vnc is recording a cxtest test(ie button clicks, keystrokes). That doesn't have to be fast, just beeing able to init glx is ok. But I agree on the non-trivial point. So far I've used manually coded keystrokes instead of button clicks, and it works, although it a bit of pain.
On 28/06/07, Stefan Dösinger [email protected] wrote:
Am Donnerstag, 28. Juni 2007 09:09 schrieb H. Verbeet:
On 28/06/07, Steven Edwards [email protected] wrote:
Could we add an extension to vnc to use Mesa?
I suspect it would be non-trivial, but more importantly I think it would defeat the purpose of measuring performance on different hardware configurations.
The problem with vnc is recording a cxtest test(ie button clicks, keystrokes). That doesn't have to be fast, just beeing able to init glx is ok. But I agree on the non-trivial point. So far I've used manually coded keystrokes instead of button clicks, and it works, although it a bit of pain.
I see. With eg. 3DMark you'd still have caps issues though.
The problem with vnc is recording a cxtest test(ie button clicks, keystrokes). That doesn't have to be fast, just beeing able to init glx is ok. But I agree on the non-trivial point. So far I've used manually coded keystrokes instead of button clicks, and it works, although it a bit of pain.
I see. With eg. 3DMark you'd still have caps issues though.
Thats true, but fortunately I don't need the whole set of tests to run while recording. Its enough to get the results dialog to continue recording from there. For that I could fake some caps temporarily in wined3d.
Stefan Dösinger wrote:
--no-vnc works fine for me for playing tests, but it does not work for recording tests.
VNC was never hard requirement of CxTest, we are using it to guarantee same testing environment everywhere. However, it is been a while I ran it without VNC, let me check.
Martin
Hello Stefan!
We don't have to transfer all the rendering screenshots over VNC though. I am CCing Martin Pilka on this, maybe he can give us some more insight.
I apologize for late answer - I was (and still am) ill. However, now I read whole thread, let me summarize the goals you would expect from test framework:
* Ability to regularly benchmark certain parts of game (and thus Wine, which is involved) in clean environment and submit raw data results to server
* On the server, process submitted data and present them via web in user friendly form; support for pointing to regressions (and finding regression patches) would be nice
Did I miss some important aspect? Feel free to comment.
I try package you created with Francois soon. Then I have a meeting with other guys who also work on CxTest, we try to find way how CxTest could meet goals we define here.
Martin
- Ability to regularly benchmark certain parts of game (and thus Wine,
which is involved) in clean environment and submit raw data results to server
- On the server, process submitted data and present them via web in user
friendly form; support for pointing to regressions (and finding regression patches) would be nice
Did I miss some important aspect? Feel free to comment.
Yes, thats essentially what we need. I've been setting up a quick and dirty test site myself, with a few lines of php and gnuplot:
http://84.112.174.163/~stefan/laptop/3dmark2000/results.php
The whole setup is highly inflexible, doesn't contain any real test data yet, just quick and dirty(~100 lines of php and ~70 lines of gnuplot script).
Stefan Dösinger wrote:
Yes, thats essentially what we need. I've been setting up a quick and dirty test site myself, with a few lines of php and gnuplot:
Stefan, could you put the link life again? Or send me screenshots of it. I am working on small specification draft what we could do in first iteration.
Thanks, Martin
On 20/12/2007, Mirek Slugeň [email protected] wrote:
OLD CONFIGURATION: Configuration: Core 2 Duo E6400@3200MHz, 2GB RAM, GF 6800 GS Options in wine registry: GLSL, FBO, VideoMemorySize 256, PS, VS, SoftwareEmulation Patches: 3DMark hack
ACTUAL CONFIGURATION (bold results): Configuration: Core 2 Duo E6400@3200MHz, 2GB RAM, GF 7900 GT (nvidia 169.04) Options in wine registry: GLSL, FBO, VideoMemorySize 256, PS, VS, SoftwareEmulation Patches: 3DMark hack, small gothic patch, wined3d: rewrite IWineD3DSurfaceImpl_BltOverride colorfill codepath, wined3d: use IWineD3DDeviceImpl_ClearSurface in IWineD3DDeviceImpl_Clear
Changing the graphics card skews the results a bit :-/ Unfortunately it's pretty hard to say how much of an impact that has per test.
On Thu, Dec 20, 2007 at 03:58:05PM +0100, H. Verbeet wrote:
OLD CONFIGURATION: Configuration: Core 2 Duo E6400@3200MHz, 2GB RAM, GF 6800 GS Options in wine registry: GLSL, FBO, VideoMemorySize 256, PS, VS, SoftwareEmulation Patches: 3DMark hack
ACTUAL CONFIGURATION (bold results): Configuration: Core 2 Duo E6400@3200MHz, 2GB RAM, GF 7900 GT (nvidia 169.04) Options in wine registry: GLSL, FBO, VideoMemorySize 256, PS, VS, SoftwareEmulation Patches: 3DMark hack, small gothic patch, wined3d: rewrite IWineD3DSurfaceImpl_BltOverride colorfill codepath, wined3d: use IWineD3DDeviceImpl_ClearSurface in IWineD3DDeviceImpl_Clear
Changing the graphics card skews the results a bit :-/ Unfortunately it's pretty hard to say how much of an impact that has per test.
and changing drivers (assumption) and the patches won't help either.
Am Donnerstag, 20. Dezember 2007 16:45:11 schrieb Christoph Frick:
Changing the graphics card skews the results a bit :-/ Unfortunately it's pretty hard to say how much of an impact that has per test.
and changing drivers (assumption) and the patches won't help either.
Yeah, the different hardware setup limits the usefulness of the results. It would be really interesting to see this comparison with the same hardware and driver :-)
On a related note, it would be really cool if people could help with writing cxtest(www.cxtest.org) scripts for the various apps. This will allow automated performance testing and automated one-click regression testing if problems occur. I have once written such benchmarks for ut2004, half life 2 and 3dmark2000, but more are definitly welcome. My time is too limited to take care of all this, so our D3D QA still relies on user testing for the most part.
On Dec 20, 2007 9:33 AM, Mirek Slugeň [email protected] wrote:
3DMark 2006 game tests: 768/126/1149 points game 1: 5.2/1.2/9.3 fps game 2: 7.6/0.9/9.7 fps hdr game tests: 298/167/1022 points game 1: 1.976/1.4/8.8 fps game 2: 3.989/1.9/11.6 fps cpu tests: 2412/2426/2665 points game 1: 0.782/0.794/0.855 fps game 2: 1.190/1.186/1.329 fps
Mirek Slugen
I ran 3DMark06 just now and got a score of 3464
Some System Specs: Quad Core 2.4Ghz, 8GB RAM, 1.2TB HD Space, 640MB 8800GTS, Gentoo Linux
SM2.0 = 1662 HDR SM3 = 880 CPU = 2841
http://img409.imageshack.us/img409/7809/screenshotwinedesktop2oy0.png http://img516.imageshack.us/img516/3826/screenshotwinedesktop3vu1.png http://img521.imageshack.us/img521/252/screenshotwinedesktop4xo2.png
Tom
On Dec 21, 2007 12:52 AM, Tom Wickline [email protected] wrote:
640MB 8800GTS, Gentoo Linux
That should say 8800GTX
Tom
Running applications with different log levels, tracing output, comparing screenshots with different levels of fuzzy comparaion
- it's not faraway.
How would fuzzy comparison work?
See projects findimagedupes and gqview (duplicate window, hit 'D' in gqview interface), both presented as packages in standard Debian. These fuzzy compare images. You can find some images which can be used as input here:
http://dl.ifne.eu/image_matching.tar.gz
Results vary, findimagedupes is quite good, however its accuracy is below 100%.
Martin
-
Christoph Frick -
Francois Gouget -
H. Verbeet -
martin pilka -
Mirek Slugeň -
Stefan Dösinger -
Stefan Dösinger -
Steven Edwards -
Tom Wickline -
Vit Hrachovy