| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-20 | |||
| 20:13:06 | efried | made it disappear from the little flow charty thingy. | |
| 20:17:35 | mriedem | i still see it here https://launchpad.net/nova/train | |
| 20:17:42 | mriedem | but not marked active https://launchpad.net/nova/+milestone/train-2 | |
| 20:17:44 | mriedem | which means, | |
| 20:18:05 | mriedem | it doesn't show up as a milestone target either, which is what we want | |
| 20:18:09 | mriedem | so we're good | |
| 20:23:03 | efried | \o/ | |
| 20:38:26 | openstackgerrit | Eric Fried proposed openstack/nova master: objects: Rename 'nova.objects.instance_numa_topology' https://review.opendev.org/671789 | |
| 20:38:26 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Remove unnecessary try-catch around 'getCPUMap' https://review.opendev.org/671790 | |
| 20:38:27 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Remove unnecessary argument https://review.opendev.org/674396 | |
| 20:38:28 | openstackgerrit | Eric Fried proposed openstack/nova master: objects: Remove legacy '_from_dict' functions https://review.opendev.org/537414 | |
| 20:38:28 | openstackgerrit | Eric Fried proposed openstack/nova master: claims: Remove useless caching https://review.opendev.org/671791 | |
| 20:38:29 | openstackgerrit | Eric Fried proposed openstack/nova master: trivial: Rename exception argument https://review.opendev.org/671795 | |
| 20:38:29 | openstackgerrit | Eric Fried proposed openstack/nova master: Remove 'hardware.get_host_numa_usage_from_instance' https://review.opendev.org/671797 | |
| 20:38:30 | openstackgerrit | Eric Fried proposed openstack/nova master: Remove 'hardware.host_topology_and_format_from_host' https://review.opendev.org/671798 | |
| 20:38:30 | openstackgerrit | Eric Fried proposed openstack/nova master: Remove 'hardware.instance_topology_from_instance' https://review.opendev.org/671799 | |
| 20:38:31 | openstackgerrit | Eric Fried proposed openstack/nova master: Rework 'hardware.numa_usage_from_instances' https://review.opendev.org/672565 | |
| 20:38:31 | openstackgerrit | Eric Fried proposed openstack/nova master: tests: Split NUMA object tests https://review.opendev.org/672336 | |
| 20:38:32 | openstackgerrit | Eric Fried proposed openstack/nova master: Add '[compute] cpu_dedicated_set' option https://review.opendev.org/671792 | |
| 20:38:32 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement https://review.opendev.org/671793 | |
| 20:38:32 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: '_get_(v|p)cpu_total' to '_get_(v|p)cpu_available' https://review.opendev.org/672693 | |
| 20:38:33 | openstackgerrit | Eric Fried proposed openstack/nova master: trivial: Rewrap definitions of 'NUMACell' https://review.opendev.org/674395 | |
| 20:38:34 | openstackgerrit | Eric Fried proposed openstack/nova master: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800 | |
| 20:38:34 | openstackgerrit | Eric Fried proposed openstack/nova master: objects: Rename 'fields' import to 'obj_fields' https://review.opendev.org/674103 | |
| 20:38:35 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Start reporting 'HW_CPU_HYPERTHREADING' trait https://review.opendev.org/675571 | |
| 20:38:46 | melwitt | mriedem: the archive command should go after 'set -e' too right? or no? | |
| 20:39:29 | melwitt | oh, it shouldn't bc we expect it to return 1 several times | |
| 20:40:54 | mriedem | melwitt: i think it can because the functions will only return a non-0 if they fail | |
| 20:41:19 | melwitt | mriedem: oh, I see. the function, not the individual commands. ok thanks | |
| 20:41:21 | mriedem | that's one issue i've meant to cleanup in that script is if the archive or purge fails the script wouldn't actually fail and the job wouldnt' fail | |
| 20:42:29 | melwitt | yeah, I see | |
| 20:42:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters https://review.opendev.org/673496 | |
| 20:45:44 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters https://review.opendev.org/673496 | |
| 20:45:48 | mriedem | sean-k-mooney: stephenfin: i've addressed comments in ^ and am +2 now | |
| 20:48:34 | openstackgerrit | melanie witt proposed openstack/nova master: Verify archive_deleted_rows --all-cells in post test hook https://review.opendev.org/672840 | |
| 20:55:39 | mriedem | efried: looking at the grenade failures in https://review.opendev.org/#/c/676684/, it appears to be segfaults killing q-svc | |
| 20:55:59 | mriedem | Aug 20 15:53:30 ubuntu-bionic-rax-dfw-0010301045 kernel: /usr/bin/python[21875]: segfault at a9 ip 000055aee57ec9e4 sp 00007ffed66329d8 error 4 in python2.7[55aee5716000+301000] | |
| 20:56:05 | mriedem | which i've seen locally but not so much in the gate | |
| 20:56:11 | efried | mriedem: segfault + switch-to-something-called-binary == suspicious | |
| 20:56:34 | mriedem | but these jobs aren't even using postgresql | |
| 20:56:50 | efried | I didn't go back and check the previous failures | |
| 20:57:02 | efried | but it seemed like *all* the grenade jobs failed *every* patch set. | |
| 20:57:08 | efried | (anecdotal ^ ) | |
| 20:58:07 | mriedem | the pips are installed | |
| 20:58:08 | mriedem | psycopg2==2.8.3 psycopg2-binary==2.8.3 | |
| 20:58:08 | efried | Anyway, I have a definite -1 on the requirements-check problem. If the grenade thing turns out to be unrelated, great. | |
| 20:58:26 | mriedem | the good news is there are hits in logstash | |
| 20:58:29 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Unexpected%20exception%20in%20API%20method%3A%20ConnectFailure%3A%20Unable%20to%20establish%20connection%20to%5C%22%20AND%20message%3A%5C%22v2.0%2Fports%3Fdevice_id%5C%22%20AND%20message%3A%5C%22('Connection%20aborted.'%2C%20BadStatusLine('No%20status%20line%20received%20-%20the%20server%20has%20closed%20the%20connection'%2C))%5C%22%20AND%20tags | |
| 20:58:30 | mriedem | 5C%22screen-n-api.txt%5C%22&from=7d | |
| 20:58:35 | mriedem | the bad news is they are only on that one change | |
| 20:58:44 | efried | is that bad news? | |
| 20:58:56 | efried | It seems like it would be further evidence that the problem is *there*. | |
| 21:02:45 | mriedem | well it's not good news | |
| 21:03:19 | mriedem | i've seen this same thing in my devstack vms i've created lately b/c there is too much filling up a single 8gb ram/8vpu vm even after stripping the services to pretty bare bone | |
| 21:03:25 | mriedem | clarkb has posted about it in the ML so it's a known issue | |
| 21:03:36 | mriedem | but not many people working on it | |
| 21:05:21 | efried | is it possible that the -binary thingy is just sucking up too much memory? | |
| 21:07:15 | my_nd | hi there, is there a way to configure a compute node to use only some % of their available resources to host instances? | |
| 21:10:40 | mriedem | my_nd: yes, you can configured reserved cpu, ram and disk for the host | |
| 21:10:58 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.reserved_host_cpus | |
| 21:11:05 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.reserved_host_disk_mb | |
| 21:11:13 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.reserved_host_memory_mb | |
| 21:12:07 | my_nd | thanks mriedem | |
| 21:13:13 | mriedem | efried: postgresql shouldn't even be running | |
| 21:13:35 | mriedem | i don't see it in https://a6b55b727208d469fce9-a76e08006e67cd388b33580b35bd0960.ssl.cf2.rackcdn.com/676684/7/check/nova-grenade-live-migration/8626716/logs/screen-peakmem_tracker.txt.gz anyway | |
| 21:13:44 | mriedem | mysqld is in there | |
| 21:16:23 | mriedem | unfortunately peakmemtracker seems to die before we start seeing segfaults | |
| 21:16:40 | mriedem | top memory usage is unsurprisingly mysqld, neutron, and nova (api and compute) | |
| 21:18:01 | efried | I can't make that logstash query work, because it split lines on me. Can you paste just the text from the query box? | |
| 21:18:23 | mriedem | message:"Unexpected exception in API method: ConnectFailure: Unable to establish connection to" AND message:"v2.0/ports?device_id" AND message:"('Connection aborted.', BadStatusLine('No status line received - the server has closed the connection',))" AND tags:"screen-n-api.txt" | |
| 21:18:29 | efried | or you can just tell me: are there hits from the earlier patch sets? | |
| 21:18:55 | mriedem | 6 and 7 yes | |
| 21:19:18 | efried | mm | |
| 21:19:37 | mriedem | nothing really changed much since PS5 though... | |
| 21:19:46 | efried | those are all from nova-grenade-live-migration. | |
| 21:20:03 | efried | how did the other grenades fail? | |
| 21:23:13 | mriedem | similar, but in one case the grenade script calling neutron api directly rather than proxying through nova | |
| 21:23:14 | mriedem | 2019-08-20 15:42:11.965 | + /opt/stack/new/grenade/projects/70_cinder/resources.sh:destroy:265 : openstack floating ip unset --port 172.24.5.75 2019-08-20 15:42:15.779 | Unable to establish connection to http://104.239.175.129:9696/v2.0/floatingips/172.24.5.75: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',)) | |
| 21:23:36 | mriedem | Aug 20 15:42:15.153161 ubuntu-bionic-rax-iad-0010301032 neutron-server[11166]: INFO oslo_service.service [-] Child 24291 killed by signal 11 | |
| 21:24:28 | mriedem | Aug 20 15:42:15 ubuntu-bionic-rax-iad-0010301032 kernel: /usr/bin/python[22305]: segfault at a9 ip 0000000000585794 sp 00007ffd8b90c5b8 error 4 in python3.6[400000+3bf000] | |
| 21:32:39 | efried | Is there a unifying logstash search to rule them all? | |
| 21:35:44 | efried | message:"Unable to establish connection to" AND tags:"screen-n-api.txt" | |
| 21:35:44 | efried | with 7d | |
| 21:35:44 | efried | gives me 64 hits on three different grenade jobs on both of those patches and no others | |
| 21:36:00 | efried | (that one and its now-abandoned successor) | |
| 21:36:24 | efried | That would be enough for me to say, "sorry, this -binary thing is no good". | |
| 21:47:25 | mriedem | yeah there were hits on the now abandoned patch that was on top of the -binary one too | |
| 21:47:26 | mriedem | super weird | |
| 21:52:02 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete InstanceMapping in conductor if BuildRequest is already deleted https://review.opendev.org/666438 | |
| 21:57:36 | mriedem | dansmith: would be nice to get this in https://review.opendev.org/#/c/673620/ so we can drop the service uuid online data migration in U | |
| 21:59:38 | dansmith | mriedem: hmm, we've had other blocker migrations since havana that didn't get added to that special list | |
| 22:00:41 | mriedem | i know, and they don't do anything | |
| 22:01:02 | dansmith | why do you need to add yours there then? | |
| 22:01:22 | mriedem | https://review.opendev.org/#/c/673620/1/nova/tests/unit/db/test_migrations.py@730 | |
| 22:01:27 | mriedem | because it's cleaner? | |
| 22:01:46 | mriedem | my guess is that skip list came after the older blocker migrations | |
| 22:01:55 | dansmith | I don't think so, | |
| 22:02:00 | dansmith | I think I added that a long time ago | |
| 22:03:31 | mriedem | ok yeah https://review.opendev.org/#/c/122162/ vs https://review.opendev.org/#/c/174480/ | |
| 22:03:36 | mriedem | but having said that, i'm not sure why it matters | |