| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-23 | |||
| 19:27:06 | donnyd | what about limestone or vexxhost? | |
| 19:27:25 | donnyd | i think limestone is setup like FN with ipv6 | |
| 19:28:08 | sean-k-mooney | this is the logstash link for elastic search http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Timed%20out%20waiting%20for%20response%20from%20cell%5C%22%20AND%20tags:%5C%22screen-n-sch.txt%5C%22%20AND%20voting:1&from=864000s | |
| 19:28:10 | donnyd | And I think maybe using the custom labels will make sure you can schedule the job to a provider known to fail.. | |
| 19:29:29 | mriedem | this would be right around the time where we hit the scheduler for a tempest test to create a server and the timeout happens 1 minute after that | |
| 19:29:29 | mriedem | https://zuul.opendev.org/t/openstack/build/d53346210978403f888b85b82b2fe0c7/log/logs/syslog.txt.gz#8269 | |
| 19:29:45 | mriedem | idk if the netlink messages in there could be an issue | |
| 19:29:54 | sean-k-mooney | mriedem: i just realised im looking at a single node greade-heat job which has the same issue | |
| 19:31:44 | openstackgerrit | Merged openstack/nova master: [Trivial]Remove unused helper should_switch_to_postcopy https://review.opendev.org/679177 | |
| 19:31:50 | openstackgerrit | Merged openstack/nova master: [Trivial]Removed unused helper _extract_query_params https://review.opendev.org/679174 | |
| 19:32:29 | mriedem | looks like those syslog messages are 'normal' | |
| 19:32:46 | dansmith | mriedem: yeah probably not related to this | |
| 19:33:19 | dansmith | I dunno if OVS uses netlink for any datapath stuff, but in general, it's for setup of networking stuff, | |
| 19:33:30 | dansmith | not related to moving actual packets | |
| 19:33:46 | dansmith | although I guess if those indicated failure to set something up, then maybe, but.. | |
| 19:33:52 | donnyd | I am also curious if this job was succeeding before I changed the MTU on FN | |
| 19:33:53 | sean-k-mooney | it uses netlink for a few things | |
| 19:34:25 | sean-k-mooney | donnyd: i dont think its MTU related | |
| 19:34:44 | donnyd | Well did this job pass on FN before 12 Sep? | |
| 19:34:45 | sean-k-mooney | as i said the logs i was looking at for the greade-heat job that failed was singlenode | |
| 19:34:49 | donnyd | or has it always failed | |
| 19:36:10 | mriedem | donnyd: this is recent | |
| 19:36:25 | mriedem | logstash only goes back 10 days for us but i'm seeing it go to at least 9/14 | |
| 19:36:36 | mriedem | i only noticed the failure while looking at some logs this weekend thouh | |
| 19:36:37 | mriedem | *though | |
| 19:36:42 | mriedem | so idk when it started, | |
| 19:36:58 | donnyd | reason I ask is the MTU on FN was changed on the 12th here https://review.opendev.org/#/c/681951/ | |
| 19:37:04 | mriedem | if you haven't noticed yet, when we get close to feature freeze and RC1, people recheck job failures like a chicken with its head cut off just to get their code merged | |
| 19:37:56 | mriedem | Sep 22 00:14:19 ubuntu-bionic-ovh-gra1-0011664420 sudo[13369]: stack : TTY=unknown ; PWD=/opt/stack/old/devstack ; USER=root ; COMMAND=/sbin/ip link set mtu 1500 dev br-ex | |
| 19:41:32 | donnyd | message:"sbin/ip link set mtu 1500 dev br-ex" AND node_provider:"fortnebula-regionone" | |
| 19:41:43 | donnyd | yea that is probably not going to work well | |
| 19:42:50 | donnyd | Sep 23 16:41:40 ubuntu-bionic-fortnebula-regionone-0011712952 sudo[14580]: stack : TTY=unknown ; PWD=/opt/stack/old/devstack ; USER=root ; COMMAND=/sbin/ip link set mtu 1500 dev br-ex | |
| 19:43:36 | openstackgerrit | Merged openstack/nova master: Remove stubs from VolumeAttachmentsSample API sample test https://review.opendev.org/680834 | |
| 19:43:42 | openstackgerrit | Merged openstack/nova master: Use multiple attachments in test_list_volume_attachments https://review.opendev.org/681618 | |
| 19:43:43 | donnyd | best case it will just cause every packet to be retransmitted at a lower MTU... | |
| 19:44:36 | sean-k-mooney | donnyd: again i dont think this is mtu related. none of the tests shoudl need to send packet that leave teh host | |
| 19:45:13 | sean-k-mooney | donnyd: the way tempest executs zuul ssh's into the vm provided by FN and runs evertying from within that vm | |
| 19:45:35 | sean-k-mooney | in a single node deployment that means that noting should leave the vm | |
| 19:45:43 | donnyd | Well I don't think they would have to... | |
| 19:45:44 | sean-k-mooney | so the MTU would cause any issues | |
| 19:46:08 | donnyd | the interface itself that br-ex is connected to would have a lower MTU though wouldn't it | |
| 19:46:50 | donnyd | and is the API connected to that same interface? | |
| 19:46:54 | sean-k-mooney | it may but tempest wont be using that interface | |
| 19:47:39 | donnyd | Ok | |
| 19:48:00 | donnyd | I think we should just use the custom label and then hold the node | |
| 19:48:20 | donnyd | so we can see what is in fact the what | |
| 19:48:58 | hemna | hrmm, so the tox -epy27 issue goes away on ubuntu for stable/rocky | |
| 19:49:10 | hemna | happens on opensuse Leap | |
| 19:49:12 | hemna | :( | |
| 19:49:20 | sean-k-mooney | what is the issue? | |
| 19:49:32 | hemna | http://paste.openstack.org/show/778953/ | |
| 19:49:35 | donnyd | yes what=issue | |
| 19:50:24 | mriedem | hemna: oh, i can't speak for suse, i only use bionic for unit test runs and local dev | |
| 19:50:59 | mriedem | maybe some sqlite issue with the suse you're using? | |
| 19:51:55 | hemna | hrmm yah I suppose so | |
| 19:52:18 | owalsh | lyarwood, mriedem: AFAIK tripleo isn't doing anything for the cells (apart from the default cell). The cell has to be manually added and the api services need to be bounced | |
| 19:53:58 | mriedem | owalsh: ack | |
| 19:54:56 | owalsh | mriedem: and as lee mentioned support is recent so there isn't an upgrade path to worry about yet. Does something need poking? | |
| 19:56:18 | mriedem | no, just started seeing some weirdness related to cell connection timeouts during scheduling after upgrade in our grenade jobs | |
| 19:57:11 | mriedem | melwitt: this should be good to go now, your comments have been addressed https://review.opendev.org/#/c/541420/ | |
| 19:57:15 | mriedem | and the patch on top is happy | |
| 19:58:44 | openstackgerrit | Merged openstack/nova master: Tune up db.instance_get_all_uuids_by_hosts https://review.opendev.org/679627 | |
| 19:58:56 | openstackgerrit | Merged openstack/nova master: Add reminder to update corresponding glance docs https://review.opendev.org/682012 | |
| 20:10:09 | efried | mriedem: docs for vpmem if you please? https://review.opendev.org/#/c/680300/ | |
| 20:17:15 | mriedem | efried: one of the cores that reviewed that series can hit it | |
| 20:17:21 | mriedem | i wouldn't know if any of it is true or not | |
| 20:17:30 | efried | ack | |
| 20:22:05 | openstackgerrit | Merged openstack/nova master: Refactor pre-live-migration work out of _do_live_migration https://review.opendev.org/641453 | |
| 20:41:35 | melwitt | mriedem: ack, will look | |
| 21:32:42 | openstackgerrit | Eric Fried proposed openstack/nova-specs master: resubmit image metadata prefiltering spec for ussuri https://review.opendev.org/683258 | |
| 21:34:40 | openstackgerrit | Merged openstack/nova-specs master: Spec: Provider config YAML file https://review.opendev.org/680471 | |
| 21:49:12 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Log error when volume validation fails during boot from volume https://review.opendev.org/684140 | |
| 21:49:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional tests for [cinder]/cross_az_attach=False https://review.opendev.org/684141 | |
| 21:49:16 | mriedem | dansmith: ^ i wrote some functional tests for that long-standing cross_az_attach api bug | |
| 21:49:23 | mriedem | rather than rely on devstack wip'ery | |
| 21:50:39 | mriedem | crap, i should actually add the file | |
| 21:50:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional tests for [cinder]/cross_az_attach=False https://review.opendev.org/684141 | |
| 22:02:03 | openstackgerrit | Merged openstack/nova-specs master: resubmit image metadata prefiltering spec for ussuri https://review.opendev.org/683258 | |
| 22:43:48 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Update keypairs in saving an instance object https://review.opendev.org/683043 | |
| #openstack-nova - 2019-09-24 | |||
| 00:22:04 | openstackgerrit | Merged openstack/nova master: Functional reproduction for bug 1844993 https://review.opendev.org/683947 | |
| 00:22:05 | openstack | bug 1844993 in OpenStack Compute (nova) "migrate a server with qos port with compute RPC pinned to 5.1 fails and leaves the qos port in an inconsistent state" [High,In progress] https://launchpad.net/bugs/1844993 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 00:28:39 | openstackgerrit | Sundar Nadathur proposed openstack/nova-specs master: Re-proposed Nova Cyborg interaction specification. https://review.opendev.org/682637 | |
| 00:28:40 | openstackgerrit | Sundar Nadathur proposed openstack/nova-specs master: Updated Nova-Cyborg interaction spec. https://review.opendev.org/684151 | |
| 02:36:23 | openstackgerrit | Merged openstack/nova master: Add librsvg2* to bindep https://review.opendev.org/683003 | |
| 02:41:12 | openstackgerrit | Takashi NATSUME proposed openstack/nova stable/rocky: Replace glance command with openstack command https://review.opendev.org/637060 | |
| 08:13:41 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject migration with QoS port from conductor if RPC pinned https://review.opendev.org/683948 | |
| 08:49:14 | openstackgerrit | Arthur Dayne proposed openstack/nova master: Fix block disk attachment failure https://review.opendev.org/682772 | |
| 08:49:53 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Remove upgrade specific info from user facing exception text https://review.opendev.org/684253 | |
| 09:05:31 | openstackgerrit | Arthur Dayne proposed openstack/nova master: Fix block disk attachment failure https://review.opendev.org/682772 | |
| 09:16:48 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Move HostNameWeigher to a common fixture https://review.opendev.org/683874 | |
| 09:59:11 | tellesnobrega | hi folks, is there a way from nova to check state of instances and update db if it detects that an instance is not running anymore? | |
| 10:00:13 | tellesnobrega | I saw the reset-state command but wanted to make sure it does what I'm looking for | |
| 11:54:57 | openstackgerrit | Jacek Tomasiak proposed openstack/nova master: Ensure volume filesystems are mounted on resume https://review.opendev.org/684290 | |
| 12:07:16 | gibi | bauzas: left comments and a bug report in https://review.opendev.org/#/c/670112 | |
| 12:11:03 | cdent | gibi: you have a theory about what's going wrong there? | |
| 12:11:47 | gibi | cdent: not yet but I can dig in if needed | |
| 12:12:07 | cdent | no, I was just curious if anything obvious stuck out. and was very happy to see you testing it with real stuff | |
| 12:12:18 | cdent | presumably better functional tests is needed | |
| 12:21:34 | gibi | cdent: I have one commment about context targeting when querying instances from the db that could be the reason | |
| 12:28:57 | gibi | cdent: yeah something with the db access is problematic as the script does not even found the compute node. I guess it querying from the wrong cell db | |