| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-20 | |||
| 20:38:46 | melwitt | mriedem: the archive command should go after 'set -e' too right? or no? | |
| 20:39:29 | melwitt | oh, it shouldn't bc we expect it to return 1 several times | |
| 20:40:54 | mriedem | melwitt: i think it can because the functions will only return a non-0 if they fail | |
| 20:41:19 | melwitt | mriedem: oh, I see. the function, not the individual commands. ok thanks | |
| 20:41:21 | mriedem | that's one issue i've meant to cleanup in that script is if the archive or purge fails the script wouldn't actually fail and the job wouldnt' fail | |
| 20:42:29 | melwitt | yeah, I see | |
| 20:42:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters https://review.opendev.org/673496 | |
| 20:45:44 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Deprecate Aggregate[Core|Ram|Disk]Filters https://review.opendev.org/673496 | |
| 20:45:48 | mriedem | sean-k-mooney: stephenfin: i've addressed comments in ^ and am +2 now | |
| 20:48:34 | openstackgerrit | melanie witt proposed openstack/nova master: Verify archive_deleted_rows --all-cells in post test hook https://review.opendev.org/672840 | |
| 20:55:39 | mriedem | efried: looking at the grenade failures in https://review.opendev.org/#/c/676684/, it appears to be segfaults killing q-svc | |
| 20:55:59 | mriedem | Aug 20 15:53:30 ubuntu-bionic-rax-dfw-0010301045 kernel: /usr/bin/python[21875]: segfault at a9 ip 000055aee57ec9e4 sp 00007ffed66329d8 error 4 in python2.7[55aee5716000+301000] | |
| 20:56:05 | mriedem | which i've seen locally but not so much in the gate | |
| 20:56:11 | efried | mriedem: segfault + switch-to-something-called-binary == suspicious | |
| 20:56:34 | mriedem | but these jobs aren't even using postgresql | |
| 20:56:50 | efried | I didn't go back and check the previous failures | |
| 20:57:02 | efried | but it seemed like *all* the grenade jobs failed *every* patch set. | |
| 20:57:08 | efried | (anecdotal ^ ) | |
| 20:58:07 | mriedem | the pips are installed | |
| 20:58:08 | mriedem | psycopg2==2.8.3 psycopg2-binary==2.8.3 | |
| 20:58:08 | efried | Anyway, I have a definite -1 on the requirements-check problem. If the grenade thing turns out to be unrelated, great. | |
| 20:58:26 | mriedem | the good news is there are hits in logstash | |
| 20:58:29 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Unexpected%20exception%20in%20API%20method%3A%20ConnectFailure%3A%20Unable%20to%20establish%20connection%20to%5C%22%20AND%20message%3A%5C%22v2.0%2Fports%3Fdevice_id%5C%22%20AND%20message%3A%5C%22('Connection%20aborted.'%2C%20BadStatusLine('No%20status%20line%20received%20-%20the%20server%20has%20closed%20the%20connection'%2C))%5C%22%20AND%20tags | |
| 20:58:30 | mriedem | 5C%22screen-n-api.txt%5C%22&from=7d | |
| 20:58:35 | mriedem | the bad news is they are only on that one change | |
| 20:58:44 | efried | is that bad news? | |
| 20:58:56 | efried | It seems like it would be further evidence that the problem is *there*. | |
| 21:02:45 | mriedem | well it's not good news | |
| 21:03:19 | mriedem | i've seen this same thing in my devstack vms i've created lately b/c there is too much filling up a single 8gb ram/8vpu vm even after stripping the services to pretty bare bone | |
| 21:03:25 | mriedem | clarkb has posted about it in the ML so it's a known issue | |
| 21:03:36 | mriedem | but not many people working on it | |
| 21:05:21 | efried | is it possible that the -binary thingy is just sucking up too much memory? | |
| 21:07:15 | my_nd | hi there, is there a way to configure a compute node to use only some % of their available resources to host instances? | |
| 21:10:40 | mriedem | my_nd: yes, you can configured reserved cpu, ram and disk for the host | |
| 21:10:58 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.reserved_host_cpus | |
| 21:11:05 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.reserved_host_disk_mb | |
| 21:11:13 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.reserved_host_memory_mb | |
| 21:12:07 | my_nd | thanks mriedem | |
| 21:13:13 | mriedem | efried: postgresql shouldn't even be running | |
| 21:13:35 | mriedem | i don't see it in https://a6b55b727208d469fce9-a76e08006e67cd388b33580b35bd0960.ssl.cf2.rackcdn.com/676684/7/check/nova-grenade-live-migration/8626716/logs/screen-peakmem_tracker.txt.gz anyway | |
| 21:13:44 | mriedem | mysqld is in there | |
| 21:16:23 | mriedem | unfortunately peakmemtracker seems to die before we start seeing segfaults | |
| 21:16:40 | mriedem | top memory usage is unsurprisingly mysqld, neutron, and nova (api and compute) | |
| 21:18:01 | efried | I can't make that logstash query work, because it split lines on me. Can you paste just the text from the query box? | |
| 21:18:23 | mriedem | message:"Unexpected exception in API method: ConnectFailure: Unable to establish connection to" AND message:"v2.0/ports?device_id" AND message:"('Connection aborted.', BadStatusLine('No status line received - the server has closed the connection',))" AND tags:"screen-n-api.txt" | |
| 21:18:29 | efried | or you can just tell me: are there hits from the earlier patch sets? | |
| 21:18:55 | mriedem | 6 and 7 yes | |
| 21:19:18 | efried | mm | |
| 21:19:37 | mriedem | nothing really changed much since PS5 though... | |
| 21:19:46 | efried | those are all from nova-grenade-live-migration. | |
| 21:20:03 | efried | how did the other grenades fail? | |
| 21:23:13 | mriedem | similar, but in one case the grenade script calling neutron api directly rather than proxying through nova | |
| 21:23:14 | mriedem | 2019-08-20 15:42:11.965 | + /opt/stack/new/grenade/projects/70_cinder/resources.sh:destroy:265 : openstack floating ip unset --port 172.24.5.75 2019-08-20 15:42:15.779 | Unable to establish connection to http://104.239.175.129:9696/v2.0/floatingips/172.24.5.75: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',)) | |
| 21:23:36 | mriedem | Aug 20 15:42:15.153161 ubuntu-bionic-rax-iad-0010301032 neutron-server[11166]: INFO oslo_service.service [-] Child 24291 killed by signal 11 | |
| 21:24:28 | mriedem | Aug 20 15:42:15 ubuntu-bionic-rax-iad-0010301032 kernel: /usr/bin/python[22305]: segfault at a9 ip 0000000000585794 sp 00007ffd8b90c5b8 error 4 in python3.6[400000+3bf000] | |
| 21:32:39 | efried | Is there a unifying logstash search to rule them all? | |
| 21:35:44 | efried | message:"Unable to establish connection to" AND tags:"screen-n-api.txt" | |
| 21:35:44 | efried | with 7d | |
| 21:35:44 | efried | gives me 64 hits on three different grenade jobs on both of those patches and no others | |
| 21:36:00 | efried | (that one and its now-abandoned successor) | |
| 21:36:24 | efried | That would be enough for me to say, "sorry, this -binary thing is no good". | |
| 21:47:25 | mriedem | yeah there were hits on the now abandoned patch that was on top of the -binary one too | |
| 21:47:26 | mriedem | super weird | |
| 21:52:02 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete InstanceMapping in conductor if BuildRequest is already deleted https://review.opendev.org/666438 | |
| 21:57:36 | mriedem | dansmith: would be nice to get this in https://review.opendev.org/#/c/673620/ so we can drop the service uuid online data migration in U | |
| 21:59:38 | dansmith | mriedem: hmm, we've had other blocker migrations since havana that didn't get added to that special list | |
| 22:00:41 | mriedem | i know, and they don't do anything | |
| 22:01:02 | dansmith | why do you need to add yours there then? | |
| 22:01:22 | mriedem | https://review.opendev.org/#/c/673620/1/nova/tests/unit/db/test_migrations.py@730 | |
| 22:01:27 | mriedem | because it's cleaner? | |
| 22:01:46 | mriedem | my guess is that skip list came after the older blocker migrations | |
| 22:01:55 | dansmith | I don't think so, | |
| 22:02:00 | dansmith | I think I added that a long time ago | |
| 22:03:31 | mriedem | ok yeah https://review.opendev.org/#/c/122162/ vs https://review.opendev.org/#/c/174480/ | |
| 22:03:36 | mriedem | but having said that, i'm not sure why it matters | |
| 22:03:49 | mriedem | it's just making sure there is a method for the migration, but the method just passes so... | |
| 22:03:53 | mriedem | the actual test is in another module | |
| 22:04:09 | dansmith | well, it's more just the principle of the case where nothing should be added to those exclusion lists in the future, and I don't think you need to | |
| 22:04:36 | mriedem | if you're +2 besides that will you fast approve if i make that change? | |
| 22:04:52 | dansmith | yeah | |
| 22:05:11 | dansmith | sorry | |
| 22:10:33 | mriedem | np, i have to rebase anyway | |
| 22:14:23 | dansmith | mriedem: you meant soon right? :) | |
| 22:15:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add blocker migration for completing services.uuid migration https://review.opendev.org/673620 | |
| 22:15:42 | mriedem | thar she blar | |
| 22:18:06 | dansmith | got it | |
| 23:04:34 | mriedem | thanks | |
| 23:05:29 | sean-k-mooney | mriedem: ack, thanks for updating the deprecation patch. i had planned to throw up a [WIP:U] patch to actully delete the thing in u but that is still future | |
| 23:05:42 | sean-k-mooney | just missed him hehe | |
| 23:41:12 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Public method to retrieve custom resource providers https://review.opendev.org/676029 | |
| 23:41:13 | openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP: Load the custom resource providers to resource tracker https://review.opendev.org/676522 | |
| #openstack-nova - 2019-08-21 | |||
| 01:15:27 | openstackgerrit | sean mooney proposed openstack/nova stable/pike: lxc: make use of filter python3 compatible https://review.opendev.org/676502 | |
| 01:43:30 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix missing rule description in HACKING.rst https://review.opendev.org/677589 | |
| 02:21:05 | openstackgerrit | Merged openstack/nova master: [Trivial]Remove unused helper filter_and_format_resource_metadata https://review.opendev.org/676038 | |
| 03:00:45 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add a hacking rule for non-existent assertions https://review.opendev.org/677596 | |
| 03:26:31 | openstackgerrit | Boxiang Zhu proposed openstack/nova master: Fix live migration break group policy simultaneously https://review.opendev.org/651969 | |
| 04:44:01 | openstackgerrit | melanie witt proposed openstack/nova master: Verify archive_deleted_rows --all-cells in post test hook https://review.opendev.org/672840 | |
| 04:44:51 | openstackgerrit | melanie witt proposed openstack/nova master: Verify archive_deleted_rows --all-cells in post test hook https://review.opendev.org/672840 | |
| 05:23:07 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add a hacking rule for non-existent assertions https://review.opendev.org/677596 | |
| 05:46:17 | openstackgerrit | Merged openstack/nova master: doc: Trivial fixes to API version history https://review.opendev.org/649561 | |