| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-23 | |||
| 19:09:00 | dansmith | haha | |
| 19:11:20 | hemna | so I still can't seem to run tox -epy27 on anything less than stable/rocky | |
| 19:11:24 | hemna | queens and pike both fail | |
| 19:11:35 | hemna | http://paste.openstack.org/show/778953/ | |
| 19:11:56 | hemna | 6444 failures, all the same type of failure. sqlalchemy.exc.NoSuchTableError: migration_tmp | |
| 19:12:11 | dansmith | hemna: clean your local directory | |
| 19:12:16 | hemna | I did | |
| 19:12:21 | donnyd | Is this job CPU bound? | |
| 19:12:22 | hemna | nuked all pyc files and .tox | |
| 19:12:27 | dansmith | er, well, maybe not with that _tmp prefix | |
| 19:12:44 | dansmith | hemna: yeah, normally that comes from stale migration pycs but I think this is something else | |
| 19:14:23 | mriedem | 2019-09-22 00:47:29.331 | Instance 44d7efdc-c048-4dca-8b4b-3d518321eddd is in cell: cell1 (8acfb79b-2e40-4e1c-bc3d-d404dac6db90) | |
| 19:14:23 | mriedem | looking a bit further between the n-sch restart and the timeout, verify_instance works | |
| 19:14:52 | mriedem | donnyd: i think any job running nova and neutron is CPU bound :) | |
| 19:15:41 | donnyd | Yea that is not a strength of FN | |
| 19:16:51 | sean-k-mooney | mriedem: i assume you have seen the https://c0c3548b65f303ef6c0e-9dc5526a72bde5cc52e2c616e6a483fd.ssl.cf5.rackcdn.com/682061/5/check/grenade-heat/79209ac/logs/screen-n-cond.txt.gz i assume those are a result of the timeout | |
| 19:17:20 | mriedem | what, NoValidHost? yes that's the user visible failure | |
| 19:17:34 | sean-k-mooney | ya | |
| 19:17:51 | mriedem | the scheduler asks placement for providers for a build request, it gives back 1, we ask the cell db for the compute node for that one by uuid, timeout and then run an empty list of hosts through the filters | |
| 19:18:12 | sean-k-mooney | donnyd: well for jobs that actully enable nested vert/kvm FN is proably faster then the qemu based jobs | |
| 19:19:44 | openstackgerrit | Eric Fried proposed openstack/nova master: doc: attaching virtual persistent memory to guests https://review.opendev.org/680300 | |
| 19:19:45 | sean-k-mooney | mriedem: ya i notice the timestamp in the conductor log was after the timeout. | |
| 19:19:47 | donnyd | sean-k-mooney: for that case.. and anything with heavy IO should fly in FN - I am curious if the jobs that are failing on FN are related in any way to ipv6 | |
| 19:20:30 | dansmith | donnyd: could be, as we might be initiating a connection when this fails, | |
| 19:20:37 | dansmith | which might be network-related | |
| 19:20:50 | mriedem | i didn't think grenade jobs were doing anything with ipv6 | |
| 19:21:33 | sean-k-mooney | donnyd: where you thinking about the reduced mtu | |
| 19:21:36 | dansmith | well, maybe not, but just saying, if something network-y is the difference, that could explain flakiness | |
| 19:22:11 | donnyd | I was thinking more about the other one where the node loses contact when the ipv6 public side network is created | |
| 19:22:43 | donnyd | I recently tried to help that out by lowering my RA's so when it does it will pick it back up in time | |
| 19:23:00 | sean-k-mooney | the vms are dual stack vms right. they get a public ipv6 and private ipv4 address | |
| 19:23:11 | donnyd | correct sean-k-mooney | |
| 19:23:35 | sean-k-mooney | in that case ill check what ip we actully use for devstack | |
| 19:23:50 | donnyd | The mtu was lowered to 1450 | |
| 19:23:58 | donnyd | not sure what the other providers are set at | |
| 19:24:03 | sean-k-mooney | we are using the ipv4 addres | |
| 19:24:13 | sean-k-mooney | 192.168.48.93 in this case | |
| 19:24:42 | openstackgerrit | Merged openstack/nova master: Get pci_devices from _list_devices https://review.opendev.org/680674 | |
| 19:24:53 | donnyd | can we replicate this job on a test instance that doesn't get kilt | |
| 19:25:20 | sean-k-mooney | we can always ask infra to hold the node | |
| 19:25:38 | donnyd | So we can go poke around and see what the real issue is.. if its just FN failing this job then it must be FN related... | |
| 19:25:49 | dansmith | donnyd: it's not just FN | |
| 19:26:01 | sean-k-mooney | donnyd: its also failing on both of the ovh clouds | |
| 19:26:07 | dansmith | and for this, it matters what is in the cell mapping, not what is in the config file, just FYI | |
| 19:26:48 | donnyd | so it only succeeds on rax then? | |
| 19:27:06 | donnyd | what about limestone or vexxhost? | |
| 19:27:25 | donnyd | i think limestone is setup like FN with ipv6 | |
| 19:28:08 | sean-k-mooney | this is the logstash link for elastic search http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Timed%20out%20waiting%20for%20response%20from%20cell%5C%22%20AND%20tags:%5C%22screen-n-sch.txt%5C%22%20AND%20voting:1&from=864000s | |
| 19:28:10 | donnyd | And I think maybe using the custom labels will make sure you can schedule the job to a provider known to fail.. | |
| 19:29:29 | mriedem | https://zuul.opendev.org/t/openstack/build/d53346210978403f888b85b82b2fe0c7/log/logs/syslog.txt.gz#8269 | |
| 19:29:29 | mriedem | this would be right around the time where we hit the scheduler for a tempest test to create a server and the timeout happens 1 minute after that | |
| 19:29:45 | mriedem | idk if the netlink messages in there could be an issue | |
| 19:29:54 | sean-k-mooney | mriedem: i just realised im looking at a single node greade-heat job which has the same issue | |
| 19:31:44 | openstackgerrit | Merged openstack/nova master: [Trivial]Remove unused helper should_switch_to_postcopy https://review.opendev.org/679177 | |
| 19:31:50 | openstackgerrit | Merged openstack/nova master: [Trivial]Removed unused helper _extract_query_params https://review.opendev.org/679174 | |
| 19:32:29 | mriedem | looks like those syslog messages are 'normal' | |
| 19:32:46 | dansmith | mriedem: yeah probably not related to this | |
| 19:33:19 | dansmith | I dunno if OVS uses netlink for any datapath stuff, but in general, it's for setup of networking stuff, | |
| 19:33:30 | dansmith | not related to moving actual packets | |
| 19:33:46 | dansmith | although I guess if those indicated failure to set something up, then maybe, but.. | |
| 19:33:52 | donnyd | I am also curious if this job was succeeding before I changed the MTU on FN | |
| 19:33:53 | sean-k-mooney | it uses netlink for a few things | |
| 19:34:25 | sean-k-mooney | donnyd: i dont think its MTU related | |
| 19:34:44 | donnyd | Well did this job pass on FN before 12 Sep? | |
| 19:34:45 | sean-k-mooney | as i said the logs i was looking at for the greade-heat job that failed was singlenode | |
| 19:34:49 | donnyd | or has it always failed | |
| 19:36:10 | mriedem | donnyd: this is recent | |
| 19:36:25 | mriedem | logstash only goes back 10 days for us but i'm seeing it go to at least 9/14 | |
| 19:36:36 | mriedem | i only noticed the failure while looking at some logs this weekend thouh | |
| 19:36:37 | mriedem | *though | |
| 19:36:42 | mriedem | so idk when it started, | |
| 19:36:58 | donnyd | reason I ask is the MTU on FN was changed on the 12th here https://review.opendev.org/#/c/681951/ | |
| 19:37:04 | mriedem | if you haven't noticed yet, when we get close to feature freeze and RC1, people recheck job failures like a chicken with its head cut off just to get their code merged | |
| 19:37:56 | mriedem | Sep 22 00:14:19 ubuntu-bionic-ovh-gra1-0011664420 sudo[13369]: stack : TTY=unknown ; PWD=/opt/stack/old/devstack ; USER=root ; COMMAND=/sbin/ip link set mtu 1500 dev br-ex | |
| 19:41:32 | donnyd | message:"sbin/ip link set mtu 1500 dev br-ex" AND node_provider:"fortnebula-regionone" | |
| 19:41:43 | donnyd | yea that is probably not going to work well | |
| 19:42:50 | donnyd | Sep 23 16:41:40 ubuntu-bionic-fortnebula-regionone-0011712952 sudo[14580]: stack : TTY=unknown ; PWD=/opt/stack/old/devstack ; USER=root ; COMMAND=/sbin/ip link set mtu 1500 dev br-ex | |
| 19:43:36 | openstackgerrit | Merged openstack/nova master: Remove stubs from VolumeAttachmentsSample API sample test https://review.opendev.org/680834 | |
| 19:43:42 | openstackgerrit | Merged openstack/nova master: Use multiple attachments in test_list_volume_attachments https://review.opendev.org/681618 | |
| 19:43:43 | donnyd | best case it will just cause every packet to be retransmitted at a lower MTU... | |
| 19:44:36 | sean-k-mooney | donnyd: again i dont think this is mtu related. none of the tests shoudl need to send packet that leave teh host | |
| 19:45:13 | sean-k-mooney | donnyd: the way tempest executs zuul ssh's into the vm provided by FN and runs evertying from within that vm | |
| 19:45:35 | sean-k-mooney | in a single node deployment that means that noting should leave the vm | |
| 19:45:43 | donnyd | Well I don't think they would have to... | |
| 19:45:44 | sean-k-mooney | so the MTU would cause any issues | |
| 19:46:08 | donnyd | the interface itself that br-ex is connected to would have a lower MTU though wouldn't it | |
| 19:46:50 | donnyd | and is the API connected to that same interface? | |
| 19:46:54 | sean-k-mooney | it may but tempest wont be using that interface | |
| 19:47:39 | donnyd | Ok | |
| 19:48:00 | donnyd | I think we should just use the custom label and then hold the node | |
| 19:48:20 | donnyd | so we can see what is in fact the what | |
| 19:48:58 | hemna | hrmm, so the tox -epy27 issue goes away on ubuntu for stable/rocky | |
| 19:49:10 | hemna | happens on opensuse Leap | |
| 19:49:12 | hemna | :( | |
| 19:49:20 | sean-k-mooney | what is the issue? | |
| 19:49:32 | hemna | http://paste.openstack.org/show/778953/ | |
| 19:49:35 | donnyd | yes what=issue | |
| 19:50:24 | mriedem | hemna: oh, i can't speak for suse, i only use bionic for unit test runs and local dev | |
| 19:50:59 | mriedem | maybe some sqlite issue with the suse you're using? | |
| 19:51:55 | hemna | hrmm yah I suppose so | |