gnuplot-beta Mailing List for gnuplot (Page 27)

On Wednesday, 8 April 2020 08:57:06 PDT daniloassis wrote:
> Hi 
> 
> I would like to know if it is possible to create 3D graphics but using
> boxes as in the example gallery of version 5.5. Is there a patch for
> version 5.2? I needed these graphics urgently for my paper.

Not in 5.2
But you could download the brand new first release candidate for 5.4

  https://sf.net/projects/gnuplot/files/gnuplot/testing/
	gnuplot-5.4.rc1.tar.gz

Let us know how it goes.

Hi 

I would like to know if it is possible to create 3D graphics but using
boxes as in the example gallery of version 5.5. Is there a patch for
version 5.2? I needed these graphics urgently for my paper.

Something like this:http://www.gnuplot.info/demo_5.3/3dboxes.html 

-- 
Danilo Renato de Assis
Departamento de Infra-Estrutura e Serviços - Bloco H
Técnico em Tecnologia da Informação.
UTFPR - Reitoria
(41) 3310-4826

You can download a source tarball for the first version 5.4 release candidate from 

    https://sf.net/projects/gnuplot/files/testing/gnuplot-5.4.rc1.tar.gz

Release notes here:

    http://gnuplot.sourceforge.net/ReleaseNotes_5_4.html

Please test, but there is no great hurry.
The final release of 5.4 can wait until it is ready.


	cheers, and happy plotting

		Ethan

On Thu, 2020-03-19 at 20:21 -0700, Ethan A Merritt wrote:
> On Thursday, 19 March 2020 15:17:21 PDT Juhász Péter wrote:
> > Hi,
> > 
> > Today I've tried to create a heatmap from a CSV-ish file that
> > contained
> > more than one blocks of data, and I've naively thought that I can
> > combine `plot ... index N` with `plot ... matrix rowheaders
> > columnheaders w image`. Admittedly, it's an edge case, but it's not
> > explicitly mentioned as illegal either.
> > 
> > Unfortunately, it doesn't work:
> > for small files it produces an empty plot with some error messages,
> > and
> > for large files it crashes.
> > 
> > The following one-liner will create a file that can be used to
> > reproduce the crash:
> > 
> > perl -le '$N=100; $,=v9; for (1..2) {print "x",qw(a b)x$N; print
> > "a",(1,2)x$N for 1..$N; print"" for 1..2}' > /tmp/foo
> > 
> > then in gnuplot,
> > 
> > plot '/tmp/foo' i 0 matrix rowheaders columnheaders w image
> > 
> > Decrease the value of $N in the one-liner to generate a file that
> > doesn't crash, for me the threshold was 26.
> > 
> > gnuplot version: 5.5 patchlevel 0 last modified 2019-12-22
> > 
> > The crash happens in graphics.c:process_image.
> > 
> > So it appears that `index` doesn't cooperate with `matrix` - I can
> > accept that this combination is not supported but then there should
> > be
> > a note about the fact in the documentation. And it shouldn't crash
> > in
> > any case.
> 
> I agree that the program should never crash just because the command
> or the data is not exactly as expected.  So there certainly is a bug.
> But I don't think it is a question of 'matrix' not recognizing
> 'index.
> 

You're right! Indeed, it's the `columnheaders` mode, not `matrix` in
general that triggers the crash.

> I think the problem is that it is not well defined what it means to
> have column headers in a file with multiple data blocks.
> 
> The program seems to think that there is a single set of column
> headers
> on the first line and does not expect them to reappear in front of
> each
> data block.  If I take the large test file generated by your perl
> jiffy and
> comment out the second row of headers, then the program seems to
> operate acceptably.

Indeed. If I omit `columnheaders`, I get a warning about missing or
undefined values and the image will contain a black row, but it doesn't
crash. With `columnheaders`, I get a crash with `index 0` but not with
`index 1`. Probably there is an off-by-one error somewhere that gets
exposed by the requirement of column headers.

> 
> I will try to figure out why an unexpect line of headers causes
> a segfault, but beyond that I don't know which is more common:
> 
>     1 line of column headers applying to the entire file
> or
>   a separate line of column headers before each data block.
> 
> Thoughts from anyone who deals with this kind of data file?

The CSV (or something-SV) format itself is not well defined, everyone
uses their own home-grown customary formats. In case of a single file
with multiple data blocks:

- from the software implementation standpoint, it makes more sense to
allow and expect one header line per file, at the top of the file,
because that's simpler to explain and simpler to parse;

- from the user experience standpoint, it's better to allow a header
line for every block: they might not even represent the same kind of
data so each block could have a different number of columns with
different headers.

I think there are examples for both in the wild.

best regards,
Peter Juhasz

> 

On Thursday, 19 March 2020 15:17:21 PDT Juhász Péter wrote:
> Hi,
> 
> Today I've tried to create a heatmap from a CSV-ish file that contained
> more than one blocks of data, and I've naively thought that I can
> combine `plot ... index N` with `plot ... matrix rowheaders
> columnheaders w image`. Admittedly, it's an edge case, but it's not
> explicitly mentioned as illegal either.
> 
> Unfortunately, it doesn't work:
> for small files it produces an empty plot with some error messages, and
> for large files it crashes.
> 
> The following one-liner will create a file that can be used to
> reproduce the crash:
> 
> perl -le '$N=100; $,=v9; for (1..2) {print "x",qw(a b)x$N; print
> "a",(1,2)x$N for 1..$N; print"" for 1..2}' > /tmp/foo
> 
> then in gnuplot,
> 
> plot '/tmp/foo' i 0 matrix rowheaders columnheaders w image
> 
> Decrease the value of $N in the one-liner to generate a file that
> doesn't crash, for me the threshold was 26.
> 
> gnuplot version: 5.5 patchlevel 0 last modified 2019-12-22
> 
> The crash happens in graphics.c:process_image.
> 
> So it appears that `index` doesn't cooperate with `matrix` - I can
> accept that this combination is not supported but then there should be
> a note about the fact in the documentation. And it shouldn't crash in
> any case.

I agree that the program should never crash just because the command
or the data is not exactly as expected.  So there certainly is a bug.
But I don't think it is a question of 'matrix' not recognizing 'index.

I think the problem is that it is not well defined what it means to
have column headers in a file with multiple data blocks.

The program seems to think that there is a single set of column headers
on the first line and does not expect them to reappear in front of each
data block.  If I take the large test file generated by your perl jiffy and
comment out the second row of headers, then the program seems to
operate acceptably.

I will try to figure out why an unexpect line of headers causes
a segfault, but beyond that I don't know which is more common:

    1 line of column headers applying to the entire file
or
  a separate line of column headers before each data block.

Thoughts from anyone who deals with this kind of data file?

	Ethan
> 
> best regards,
> Peter Juhasz

Hi,

Today I've tried to create a heatmap from a CSV-ish file that contained
more than one blocks of data, and I've naively thought that I can
combine `plot ... index N` with `plot ... matrix rowheaders
columnheaders w image`. Admittedly, it's an edge case, but it's not
explicitly mentioned as illegal either.

Unfortunately, it doesn't work:
for small files it produces an empty plot with some error messages, and
for large files it crashes.

The following one-liner will create a file that can be used to
reproduce the crash:

perl -le '$N=100; $,=v9; for (1..2) {print "x",qw(a b)x$N; print
"a",(1,2)x$N for 1..$N; print"" for 1..2}' > /tmp/foo

then in gnuplot,

plot '/tmp/foo' i 0 matrix rowheaders columnheaders w image

Decrease the value of $N in the one-liner to generate a file that
doesn't crash, for me the threshold was 26.

gnuplot version: 5.5 patchlevel 0 last modified 2019-12-22

The crash happens in graphics.c:process_image.

So it appears that `index` doesn't cooperate with `matrix` - I can
accept that this combination is not supported but then there should be
a note about the fact in the documentation. And it shouldn't crash in
any case.

best regards,
Peter Juhasz

Ethan A Merritt <me...@uw...> writes:

> Your revised code works well in all the tests that I ran, but I am
> sure that there are people with use-cases I didn't test. If you are
> reading this and find a problem with the new code, let us know.

OK, great to hear. I'm going to use these patches in my day-to-day to
see if I hit any issues. Will post here if I do.

Thanks.

On Saturday, 14 March 2020 23:12:24 PDT Dima Kogan wrote:
> Hi.
> 
> I've been working on fixing broken "pause mouse close" behavior on the
> x11 terminal while x-forwarding. This is
> 
>   https://sourceforge.net/p/gnuplot/bugs/2208/
> 
> In the process I discovered that I already encountered weirdness in this
> area previously:
> 
>   https://sourceforge.net/p/gnuplot/mailman/message/36440354/
> 
> I just fixed all the issues, I think. These are in 2 patches on this
> branch:
> 
>   https://github.com/dkogan/gnuplot/commits/fixed-x11-waitforinput
> 
> The first patch improves diagnostic printing in this area. Instead of
> printing enums as NUMBERS, it prints the enum NAMES. This is infinitely
> more useful.
> 
> The second patch contains the actual fixes. It looks more invasive than
> it is. 99% of the modifications are in X11_waitforinput(). And much of
> it is indentation.
> 
> There were several issues, all of them caused by hoaky input
> multiplexing. I'm attaching two very simple gnuplot scripts that tickled
> the bug. To observe the problems you need to slow down the X connection
> by ssh-ing to any server, and trying to plot through x-forwarding.
> Before any of the fixes this would happen:
> 
>     $ gnuplot < mild.pause.mouse.close.broken.gp
> 
>     gnuplot> ause mouse close
>              ^
>              line 0: invalid command
> 

> [...snip...]

> It works for me so far, but this is an error-prone area. Can somebody
> take a look? The new approach is much better, I think, but more testing
> would be good.

It has been a long time since I looked at the x11 input multiplexing code
but I remember that it was a mess and many observed problems were
dealt with by slapping a special case in rather than rethinking the design.
So yes, a re-write is a good thing.

Your revised code works well in all the tests that I ran, but I am sure that
there are people with use-cases I didn't test.  If you are reading this and
find a problem with the new code, let us know.

Anyhow, I've imported your fixes into the main code repository.
Keep up the good work. 
	
	Ethan

Ethan Merritt <eam...@gm...> writes:

> Can you send me an idiot's guide to how to extract those patches from
> the github site?   I can view them as html pages, but cannot figure out
> how to download in a form I can feed to `patch`.
>
> Alternatively can you tell me, still a naive git user, how to fetch or
> cherry-pick these commits from your git instance?

Sure

  cd ethans_gnuplot_directory
  git remote add dima gi...@gi...:dkogan/gnuplot.git
  git fetch dima

The patches are now reference-able as

  dima/fixed-x11-waitforinput^

and

  dima/fixed-x11-waitforinput

You can 'git cherry-pick' these, or whatever else you want to do.

On Saturday, 14 March 2020 23:12:24 PDT Dima Kogan wrote:
> Hi.
> 
> I've been working on fixing broken "pause mouse close" behavior on the
> x11 terminal while x-forwarding. This is
> 
>   https://sourceforge.net/p/gnuplot/bugs/2208/
> 
> In the process I discovered that I already encountered weirdness in this
> area previously:
> 
>   https://sourceforge.net/p/gnuplot/mailman/message/36440354/
> 
> I just fixed all the issues, I think. These are in 2 patches on this
> branch:
> 
>   https://github.com/dkogan/gnuplot/commits/fixed-x11-waitforinput

Can you send me an idiot's guide to how to extract those patches from
the github site?   I can view them as html pages, but cannot figure out
how to download in a form I can feed to `patch`. 

Alternatively can you tell me, still a naive git user, how to fetch or
cherry-pick these commits from your git instance?

	thanks,

		Ethan


> The first patch improves diagnostic printing in this area. Instead of
> printing enums as NUMBERS, it prints the enum NAMES. This is infinitely
> more useful.
> 
> The second patch contains the actual fixes. It looks more invasive than
> it is. 99% of the modifications are in X11_waitforinput(). And much of
> it is indentation.
> 
> There were several issues, all of them caused by hoaky input
> multiplexing. I'm attaching two very simple gnuplot scripts that tickled
> the bug. To observe the problems you need to slow down the X connection
> by ssh-ing to any server, and trying to plot through x-forwarding.
> Before any of the fixes this would happen:
> 
>     $ gnuplot < mild.pause.mouse.close.broken.gp
> 
>     gnuplot> ause mouse close
>              ^
>              line 0: invalid command
> 
> This was caused by this block of code:
> 
>     if (IPC_LOCK) {
> #ifdef HAVE_USLEEP
> 	usleep(100);
> #endif
>         if (repeat_count++ < 10000)
> 	    goto AGAIN;
>     }
> 
> The slower link means that 10000 is too low. Bumping it up fixes the
> problem for this data file. But if you then run the other test file, you
> still see issues:
> 
>     $ gnuplot < hard.pause.mouse.close.broken.gp
> 
>     gnuplot> splse mouse close
>              ^
>              line 0: invalid command
> 
> This is an entirely different bug. I chased it down a few days ago, so I
> don't remember the exact details. The gist of the issue was that we
> would read some commands from ipc_back_fd then we would read some stuff
> from stdin, and then we would go back, and read the rest of the stuff
> from ipc_back_fd. This non-atomicity would break stuff. The fix was to
> read ALL the available ipc_back_fd data when possible, BEFORE reading
> anything from stdin.
> 
> The existing code was hoaky and mysterious. There were weird sleeps all
> over the place, and lots of state. The patch here refactors
> X11_waitforinput() to get rid of the sleeps and unneeded loops. The
> IPC_LOCK is gone, and is now another possible option argument to
> X11_waitforinput(). And there're better comments, I think.
> 
> It works for me so far, but this is an error-prone area. Can somebody
> take a look? The new approach is much better, I think, but more testing
> would be good.
> 
> Thanks.
> 
> 

Hi.

I've been working on fixing broken "pause mouse close" behavior on the
x11 terminal while x-forwarding. This is

  https://sourceforge.net/p/gnuplot/bugs/2208/

In the process I discovered that I already encountered weirdness in this
area previously:

  https://sourceforge.net/p/gnuplot/mailman/message/36440354/

I just fixed all the issues, I think. These are in 2 patches on this
branch:

  https://github.com/dkogan/gnuplot/commits/fixed-x11-waitforinput

The first patch improves diagnostic printing in this area. Instead of
printing enums as NUMBERS, it prints the enum NAMES. This is infinitely
more useful.

The second patch contains the actual fixes. It looks more invasive than
it is. 99% of the modifications are in X11_waitforinput(). And much of
it is indentation.

There were several issues, all of them caused by hoaky input
multiplexing. I'm attaching two very simple gnuplot scripts that tickled
the bug. To observe the problems you need to slow down the X connection
by ssh-ing to any server, and trying to plot through x-forwarding.
Before any of the fixes this would happen:

    $ gnuplot < mild.pause.mouse.close.broken.gp

    gnuplot> ause mouse close
             ^
             line 0: invalid command

This was caused by this block of code:

    if (IPC_LOCK) {
#ifdef HAVE_USLEEP
	usleep(100);
#endif
        if (repeat_count++ < 10000)
	    goto AGAIN;
    }

The slower link means that 10000 is too low. Bumping it up fixes the
problem for this data file. But if you then run the other test file, you
still see issues:

    $ gnuplot < hard.pause.mouse.close.broken.gp

    gnuplot> splse mouse close
             ^
             line 0: invalid command

This is an entirely different bug. I chased it down a few days ago, so I
don't remember the exact details. The gist of the issue was that we
would read some commands from ipc_back_fd then we would read some stuff
from stdin, and then we would go back, and read the rest of the stuff
from ipc_back_fd. This non-atomicity would break stuff. The fix was to
read ALL the available ipc_back_fd data when possible, BEFORE reading
anything from stdin.

The existing code was hoaky and mysterious. There were weird sleeps all
over the place, and lots of state. The patch here refactors
X11_waitforinput() to get rid of the sleeps and unneeded loops. The
IPC_LOCK is gone, and is now another possible option argument to
X11_waitforinput(). And there're better comments, I think.

It works for me so far, but this is an error-prone area. Can somebody
take a look? The new approach is much better, I think, but more testing
would be good.

Thanks.

Am 07.03.20 um 00:58 schrieb Ethan A Merritt:
> On Wednesday, 4 March 2020 12:34:45 PST Allin Cottrell wrote:
>> On Wed, 4 Mar 2020, Manfred Schwarb wrote:
>>
>>> More to the topic, I think the shape format is proprietary,
>>> complicated, spread over multiple files, limited (number of
>>> fields, length of field names, file size,...), so in one word:
>>> "legacy".
>>>
>>> So I don't think it is an appropriate format for interfacing with
>>> gnuplot. I made positive experiences with GeoJSON, it is a simple,
>>> very versatile format, and I think most GIS programs can read and
>>> write this format, so why not use some geojson routine and put it
>>> into gnuplot?
>>
>> I'm all in favor of nicer and non-proprietary formats, but what's
>> the comparison in terms of availability of files representing
>> polygons one might want (US states, counties or commuting zones, EU
>> countries or regions, etc.)? It's my impression that you can easily
>> pick up shapefiles for any/all of these -- nasty as the format may
>> be -- but what about GeoJSON files?
>>
>> Allin Cottrell

Allin, of course you are right, I guess there are almost no public GeoJSON
polygon files available.
GeoJSON is more of a intermediate format to transfer data from one application
to another one.

For me the choice of a pure shapefile solution simply sounds a bit wierd,
either a very slim solution supporting some simple format or a full-fledged
solution using libgdal (which supports ~50 different formats) would make sense.

>
> Good call.  Geojson looks quite promising.
> A real parser should be easy, but even a quick pass through sed
> or a text editor makes it usable.
> I have posted a proof-of-principle example using US state boundary
> data from a sample file on the geojson development site:
>
>   http://skuld.bmsc.washington.edu/people/merritt/gnuplot/
>

Thanks for the example.
Often the issue is, that the provided shapefiles are a complete mess and you have to
extract some polygons from it. With geojson, at least with the variant that gdal writes,
this is very simple: Each polygon is on its own line.
So, assuming the polygons are annotated, you can grep for a polygon and re-add header and
footer afterwards.

> 	Ethan
>
>
>
>
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> Membership management via: https://lists.sourceforge.net/lists/listinfo/gnuplot-beta
>

On Wednesday, 4 March 2020 12:34:45 PST Allin Cottrell wrote:
> On Wed, 4 Mar 2020, Manfred Schwarb wrote:
> 
> > More to the topic, I think the shape format is proprietary, 
> > complicated, spread over multiple files, limited (number of 
> > fields, length of field names, file size,...), so in one word: 
> > "legacy".
> >
> > So I don't think it is an appropriate format for interfacing with 
> > gnuplot. I made positive experiences with GeoJSON, it is a simple, 
> > very versatile format, and I think most GIS programs can read and 
> > write this format, so why not use some geojson routine and put it 
> > into gnuplot?
> 
> I'm all in favor of nicer and non-proprietary formats, but what's 
> the comparison in terms of availability of files representing 
> polygons one might want (US states, counties or commuting zones, EU 
> countries or regions, etc.)? It's my impression that you can easily 
> pick up shapefiles for any/all of these -- nasty as the format may 
> be -- but what about GeoJSON files?
> 
> Allin Cottrell

Good call.  Geojson looks quite promising.
A real parser should be easy, but even a quick pass through sed
or a text editor makes it usable.
I have posted a proof-of-principle example using US state boundary
data from a sample file on the geojson development site:

  http://skuld.bmsc.washington.edu/people/merritt/gnuplot/

	Ethan

On Wed, 4 Mar 2020, Manfred Schwarb wrote:

> Am 04.03.20 um 16:07 schrieb Allin Cottrell:
>> On Wed, 4 Mar 2020, Manfred Schwarb wrote:
>>
>>>> This may be putting the cart before the horse, but...
>>>> Do you think it would be helpful to teach gnuplot to read shape files?
>>>> I imagine it would be possible to test for the presence of shapelib.so
>>>> and provide a binary input mode:
>>>>
>>>>    splot <data> binary filetype=shapelib  with polygons
>>>>
>>>> I have never worked with this libary so I don't know if it is organized
>>>> in such a way that this would be possible.
>>>
>>> I do it using the gdal software and some shell magic:
>>>   # ogr2ogr -f "GMT" gaga.gmt gaga.shp
>>>   # grep -v "^#" gaga.gmt | sed 's/>//' > gaga.txt
>>
>> That's also the approach taken by Bob Mesibov at
>> https://www.datafix.com.au/BASHing/2018-10-31.html
>
> Well, you don't need 2 blank lines for separation, a single blank 
> line is enough.

OK, but not a big deal.

> More to the topic, I think the shape format is proprietary, 
> complicated, spread over multiple files, limited (number of 
> fields, length of field names, file size,...), so in one word: 
> "legacy".
>
> So I don't think it is an appropriate format for interfacing with 
> gnuplot. I made positive experiences with GeoJSON, it is a simple, 
> very versatile format, and I think most GIS programs can read and 
> write this format, so why not use some geojson routine and put it 
> into gnuplot?

I'm all in favor of nicer and non-proprietary formats, but what's 
the comparison in terms of availability of files representing 
polygons one might want (US states, counties or commuting zones, EU 
countries or regions, etc.)? It's my impression that you can easily 
pick up shapefiles for any/all of these -- nasty as the format may 
be -- but what about GeoJSON files?

Allin Cottrell

Am 04.03.20 um 16:07 schrieb Allin Cottrell:
> On Wed, 4 Mar 2020, Manfred Schwarb wrote:
>
>>> This may be putting the cart before the horse, but...
>>> Do you think it would be helpful to teach gnuplot to read shape files?
>>> I imagine it would be possible to test for the presence of shapelib.so
>>> and provide a binary input mode:
>>>
>>>    splot <data> binary filetype=shapelib  with polygons
>>>
>>> I have never worked with this libary so I don't know if it is organized
>>> in such a way that this would be possible.
>>
>> I do it using the gdal software and some shell magic:
>>   # ogr2ogr -f "GMT" gaga.gmt gaga.shp
>>   # grep -v "^#" gaga.gmt | sed 's/>//' > gaga.txt
>
> That's also the approach taken by Bob Mesibov at
> https://www.datafix.com.au/BASHing/2018-10-31.html
>

Well, you don't need 2 blank lines for separation, a single blank line is enough.

More to the topic, I think the shape format is proprietary, complicated, spread over
multiple files, limited (number of fields, length of field names, file size,...),
so in one word: "legacy".
So I don't think it is an appropriate format for interfacing with gnuplot.
I made positive experiences with GeoJSON, it is a simple, very versatile format,
and I think most GIS programs can read and write this format, so why not
use some geojson routine and put it into gnuplot?

Cheers,
Manfred


> I'm hopeful that libshape can do the job without requiring the whole
> of gdal. Libshape was developed by Frank Warmerdam, the guy who
> initially developed gdal, and is used by gdal's OGR library.
>
> Allin Cottrell
>

On Wed, 4 Mar 2020, Manfred Schwarb wrote:

> > This may be putting the cart before the horse, but...
> > Do you think it would be helpful to teach gnuplot to read shape files?
> > I imagine it would be possible to test for the presence of shapelib.so
> > and provide a binary input mode:
> >
> >    splot <data> binary filetype=shapelib  with polygons
> >
> > I have never worked with this libary so I don't know if it is organized
> > in such a way that this would be possible.
>
> I do it using the gdal software and some shell magic:
>   # ogr2ogr -f "GMT" gaga.gmt gaga.shp
>   # grep -v "^#" gaga.gmt | sed 's/>//' > gaga.txt

That's also the approach taken by Bob Mesibov at
https://www.datafix.com.au/BASHing/2018-10-31.html

I'm hopeful that libshape can do the job without requiring the whole
of gdal. Libshape was developed by Frank Warmerdam, the guy who
initially developed gdal, and is used by gdal's OGR library.

Allin Cottrell

>
> This may be putting the cart before the horse, but...
> Do you think it would be helpful to teach gnuplot to read shape files?
> I imagine it would be possible to test for the presence of shapelib.so
> and provide a binary input mode:
>
>    splot <data> binary filetype=shapelib  with polygons
>
> I have never worked with this libary so I don't know if it is organized
> in such a way that this would be possible.


I do it using the gdal software and some shell magic:
  # ogr2ogr -f "GMT" gaga.gmt gaga.shp
  # grep -v "^#" gaga.gmt | sed 's/>//' > gaga.txt

HTH,
Manfred


>
> A quick look at the spec for the shapefile format makes it look horrible
> (mixed little-ending big-endian in the same file?! variable record types
> that are required not to vary?!),  but if it is hidden by a usable library ....
>
> 	Ethan
>
>
>
>
>
>
>
> _______________________________________________
> gnuplot-beta mailing list
> gnu...@li...
> Membership management via: https://lists.sourceforge.net/lists/listinfo/gnuplot-beta
>

On Tue, 3 Mar 2020, Ethan A Merritt wrote:

> On Tuesday, 3 March 2020 17:05:40 PST Allin Cottrell wrote:
>> On Tue, 3 Mar 2020, Ethan A Merritt wrote:
>>
>>> On Tuesday, 3 March 2020 14:30:40 PST Allin Cottrell wrote:
>>>> This question has been posed before (a few years back) but I'm
>>>> hoping someone may be able to point me to the state-of-the-art,
>>>> if there is such.
>>>>
>>>> I'd like to be able to produce via gnuplot a map of (for example)
>>>> the USA showing the states colored by median income, or PM2.5
>>>> particulate emissions, or whatever.
>>>>
>>>> I understand that the starting point would be a suitable US states
>>>> shapefile (I know where I can get that), which would have to be
>>>> translated to a format (resembling world.dat, I suppose) that can be
>>>> understood by gnuplot. I have some notion of how that might be done.
>>>>
>>>> The next issue would be to get gnuplot to colorize specified areas
>>>> of the map according to the values of some additional data.
>>>> Unfortunately I have little notion of how to do that.
>>>
>>> The development version of gnuplot supports
>>>
>>> 	splot <data> with polygons
>>
>> Thanks Ethan, and also Dima in this thread. This sounds promising!
>> Right now I'm not sure when I'll find time to pursue this in depth,
>> but I'm definitely interested and will push it when I can.
>>
>> Allin Cottrell
>
> This may be putting the cart before the horse, but...
> Do you think it would be helpful to teach gnuplot to read shape files?

I'm sure it would be helpful...

> I imagine it would be possible to test for the presence of shapelib.so
> and provide a binary input mode:
>
>   splot <data> binary filetype=shapelib  with polygons
>
> I have never worked with this libary so I don't know if it is organized
> in such a way that this would be possible.

but I'm not yet clear on that point myself. I've taken a quick look 
at shapelib-1.5.0 (which configures and compiles without fuss on my 
machine) and seen that it can convert both .shp and .dbf files to 
what look like comprehensible text formats that "ought to be" 
usable (shp -> polygons that could go into CSV quite easily, and dbf 
-> descriptions of the elements of the shp file in their order of
appearance).

But when I said I "had some notion" of how shapelib could help, that 
wasn't an understatement! Perhaps better to say I "had an inkling" 
of how it could help.

I aim to follow up, but I'll be offline next week so I won't get 
back to this till the week after.

Allin

On Tuesday, 3 March 2020 17:05:40 PST Allin Cottrell wrote:
> On Tue, 3 Mar 2020, Ethan A Merritt wrote:
> 
> > On Tuesday, 3 March 2020 14:30:40 PST Allin Cottrell wrote:
> >> This question has been posed before (a few years back) but I'm
> >> hoping someone may be able to point me to the state-of-the-art,
> >> if there is such.
> >>
> >> I'd like to be able to produce via gnuplot a map of (for example)
> >> the USA showing the states colored by median income, or PM2.5
> >> particulate emissions, or whatever.
> >>
> >> I understand that the starting point would be a suitable US states
> >> shapefile (I know where I can get that), which would have to be
> >> translated to a format (resembling world.dat, I suppose) that can be
> >> understood by gnuplot. I have some notion of how that might be done.
> >>
> >> The next issue would be to get gnuplot to colorize specified areas
> >> of the map according to the values of some additional data.
> >> Unfortunately I have little notion of how to do that.
> >
> > The development version of gnuplot supports
> >
> > 	splot <data> with polygons
> 
> Thanks Ethan, and also Dima in this thread. This sounds promising!
> Right now I'm not sure when I'll find time to pursue this in depth, 
> but I'm definitely interested and will push it when I can.
> 
> Allin Cottrell

This may be putting the cart before the horse, but...
Do you think it would be helpful to teach gnuplot to read shape files?
I imagine it would be possible to test for the presence of shapelib.so
and provide a binary input mode:

   splot <data> binary filetype=shapelib  with polygons

I have never worked with this libary so I don't know if it is organized
in such a way that this would be possible.

A quick look at the spec for the shapefile format makes it look horrible
(mixed little-ending big-endian in the same file?! variable record types
that are required not to vary?!),  but if it is hidden by a usable library ....

	Ethan
	

On Tue, 3 Mar 2020, Ethan A Merritt wrote:

> On Tuesday, 3 March 2020 14:30:40 PST Allin Cottrell wrote:
>> This question has been posed before (a few years back) but I'm
>> hoping someone may be able to point me to the state-of-the-art,
>> if there is such.
>>
>> I'd like to be able to produce via gnuplot a map of (for example)
>> the USA showing the states colored by median income, or PM2.5
>> particulate emissions, or whatever.
>>
>> I understand that the starting point would be a suitable US states
>> shapefile (I know where I can get that), which would have to be
>> translated to a format (resembling world.dat, I suppose) that can be
>> understood by gnuplot. I have some notion of how that might be done.
>>
>> The next issue would be to get gnuplot to colorize specified areas
>> of the map according to the values of some additional data.
>> Unfortunately I have little notion of how to do that.
>
> The development version of gnuplot supports
>
> 	splot <data> with polygons

Thanks Ethan, and also Dima in this thread. This sounds promising!
Right now I'm not sure when I'll find time to pursue this in depth, 
but I'm definitely interested and will push it when I can.

Allin Cottrell

On Tuesday, 3 March 2020 15:47:32 PST Dima Kogan wrote:
> Ethan: do you have any data and sample scripts that plot real-world
> geographic polygons? If it's now a pain, I'd like to try it out.

I wish I did, for demo purposes if nothing else.
My time is spent in the atomic coordinate space of molecular modeling,
not the latitude/longitude space of earthly geography.
If Allin puts together a space-description file I'd be happy to
work up some demo scripts that use it.

	Ethan

Ethan: do you have any data and sample scripts that plot real-world
geographic polygons? If it's now a pain, I'd like to try it out.

On Tuesday, 3 March 2020 14:30:40 PST Allin Cottrell wrote:
> This question has been posed before (a few years back) but I'm 
> hoping someone may be able to point me to the state-of-the-art,
> if there is such.
> 
> I'd like to be able to produce via gnuplot a map of (for example) 
> the USA showing the states colored by median income, or PM2.5 
> particulate emissions, or whatever.
> 
> I understand that the starting point would be a suitable US states 
> shapefile (I know where I can get that), which would have to be 
> translated to a format (resembling world.dat, I suppose) that can be 
> understood by gnuplot. I have some notion of how that might be done.
> 
> The next issue would be to get gnuplot to colorize specified areas 
> of the map according to the values of some additional data. 
> Unfortunately I have little notion of how to do that.

The development version of gnuplot supports 

	splot <data> with polygons

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
gnuplot> help polygons

      splot DATA {using x:y:z} with polygons
            {fillstyle <fillstyle spec>}
            {fillcolor <colorspec>}
 `splot with polygons` uses pm3d to render individual triangles, quadrangles,
 and larger polygons in 3D.  These may be facets of a 3D surface or isolated
 shapes. The code assumes that the vertices lie in a plane.
 Vertices defining individual polygons are read from successive records of the
 input file.  A blank line separates one polygon from the next.

 The fill style and color may be specified in the splot command, otherwise the
 global fillstyle from `set style fill` is used.  Due to limitations in the
 pm3d code, a single border line style from `set pm3d border` is applied to all
 polygons.  This restriction may be removed in a later gnuplot version.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% 

This plot mode will be in version 5.4 (release candidate coming soonish).
Meanwhile you can build from the source for either branch:
  master
  branch-5-4-stable

I realize you don't need 3D, but plotting in projection is easy, and you even
have the option to map it onto a globe :-) 

I am uncertain whether there is adequate flexibility to use the 3rd coordinate
as a color flag.  If there isn't, there probably should be so let me know and 
I'll work on it.    

	Ethan

This MAY answer your question. I've done this sort of visualization
before by grabbing raster map images from any tile server (openstreetmap
for instance), and plotting data on top of the image.

Script to

- download the tiles you want
- stitch them together into one image you will be overlaying
- generate a template gnuplot script to plot this image

is available here:

  https://github.com/dkogan/osmgnuplot

The script plots lat/lon on one set of axes, and image pixel coords on
the other set. The mapping isn't linear, and the generated script
handles this. So you can plot your overlay with latlon coords. Works
great. Some (very simple) results in these blog posts:

  http://notes.secretsauce.net/notes/2017/09/25_pole-of-road-inaccessibility-of-the-contiguous-us.html

  http://notes.secretsauce.net/notes/2015/08/16_least-convenient-location-in-los-angeles-from-koreatown.html

This question has been posed before (a few years back) but I'm 
hoping someone may be able to point me to the state-of-the-art,
if there is such.

I'd like to be able to produce via gnuplot a map of (for example) 
the USA showing the states colored by median income, or PM2.5 
particulate emissions, or whatever.

I understand that the starting point would be a suitable US states 
shapefile (I know where I can get that), which would have to be 
translated to a format (resembling world.dat, I suppose) that can be 
understood by gnuplot. I have some notion of how that might be done.

The next issue would be to get gnuplot to colorize specified areas 
of the map according to the values of some additional data. 
Unfortunately I have little notion of how to do that.

Please don't point me to R. I know this sort of thing can be done 
via R packages -- with the help of hundreds of megabytes of 
additional GIS software! But it seems to me it should be doable much 
more simply, perhaps using the shapelib library (a few hundred 
Kbytes in size), if I could just figure out how to do a heatmap-type 
thing but filling irregular polygons rather than just rectangles.

-- 
Allin Cottrell
Department of Economics
Wake Forest University