| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-23 | |||
| 19:36:10 | mriedem | donnyd: this is recent | |
| 19:36:25 | mriedem | logstash only goes back 10 days for us but i'm seeing it go to at least 9/14 | |
| 19:36:36 | mriedem | i only noticed the failure while looking at some logs this weekend thouh | |
| 19:36:37 | mriedem | *though | |
| 19:36:42 | mriedem | so idk when it started, | |
| 19:36:58 | donnyd | reason I ask is the MTU on FN was changed on the 12th here https://review.opendev.org/#/c/681951/ | |
| 19:37:04 | mriedem | if you haven't noticed yet, when we get close to feature freeze and RC1, people recheck job failures like a chicken with its head cut off just to get their code merged | |
| 19:37:56 | mriedem | Sep 22 00:14:19 ubuntu-bionic-ovh-gra1-0011664420 sudo[13369]: stack : TTY=unknown ; PWD=/opt/stack/old/devstack ; USER=root ; COMMAND=/sbin/ip link set mtu 1500 dev br-ex | |
| 19:41:32 | donnyd | message:"sbin/ip link set mtu 1500 dev br-ex" AND node_provider:"fortnebula-regionone" | |
| 19:41:43 | donnyd | yea that is probably not going to work well | |
| 19:42:50 | donnyd | Sep 23 16:41:40 ubuntu-bionic-fortnebula-regionone-0011712952 sudo[14580]: stack : TTY=unknown ; PWD=/opt/stack/old/devstack ; USER=root ; COMMAND=/sbin/ip link set mtu 1500 dev br-ex | |
| 19:43:36 | openstackgerrit | Merged openstack/nova master: Remove stubs from VolumeAttachmentsSample API sample test https://review.opendev.org/680834 | |
| 19:43:42 | openstackgerrit | Merged openstack/nova master: Use multiple attachments in test_list_volume_attachments https://review.opendev.org/681618 | |
| 19:43:43 | donnyd | best case it will just cause every packet to be retransmitted at a lower MTU... | |
| 19:44:36 | sean-k-mooney | donnyd: again i dont think this is mtu related. none of the tests shoudl need to send packet that leave teh host | |
| 19:45:13 | sean-k-mooney | donnyd: the way tempest executs zuul ssh's into the vm provided by FN and runs evertying from within that vm | |
| 19:45:35 | sean-k-mooney | in a single node deployment that means that noting should leave the vm | |
| 19:45:43 | donnyd | Well I don't think they would have to... | |
| 19:45:44 | sean-k-mooney | so the MTU would cause any issues | |
| 19:46:08 | donnyd | the interface itself that br-ex is connected to would have a lower MTU though wouldn't it | |
| 19:46:50 | donnyd | and is the API connected to that same interface? | |
| 19:46:54 | sean-k-mooney | it may but tempest wont be using that interface | |
| 19:47:39 | donnyd | Ok | |
| 19:48:00 | donnyd | I think we should just use the custom label and then hold the node | |
| 19:48:20 | donnyd | so we can see what is in fact the what | |
| 19:48:58 | hemna | hrmm, so the tox -epy27 issue goes away on ubuntu for stable/rocky | |
| 19:49:10 | hemna | happens on opensuse Leap | |
| 19:49:12 | hemna | :( | |
| 19:49:20 | sean-k-mooney | what is the issue? | |
| 19:49:32 | hemna | http://paste.openstack.org/show/778953/ | |
| 19:49:35 | donnyd | yes what=issue | |
| 19:50:24 | mriedem | hemna: oh, i can't speak for suse, i only use bionic for unit test runs and local dev | |
| 19:50:59 | mriedem | maybe some sqlite issue with the suse you're using? | |
| 19:51:55 | hemna | hrmm yah I suppose so | |
| 19:52:18 | owalsh | lyarwood, mriedem: AFAIK tripleo isn't doing anything for the cells (apart from the default cell). The cell has to be manually added and the api services need to be bounced | |
| 19:53:58 | mriedem | owalsh: ack | |
| 19:54:56 | owalsh | mriedem: and as lee mentioned support is recent so there isn't an upgrade path to worry about yet. Does something need poking? | |
| 19:56:18 | mriedem | no, just started seeing some weirdness related to cell connection timeouts during scheduling after upgrade in our grenade jobs | |
| 19:57:11 | mriedem | melwitt: this should be good to go now, your comments have been addressed https://review.opendev.org/#/c/541420/ | |
| 19:57:15 | mriedem | and the patch on top is happy | |
| 19:58:44 | openstackgerrit | Merged openstack/nova master: Tune up db.instance_get_all_uuids_by_hosts https://review.opendev.org/679627 | |
| 19:58:56 | openstackgerrit | Merged openstack/nova master: Add reminder to update corresponding glance docs https://review.opendev.org/682012 | |
| 20:10:09 | efried | mriedem: docs for vpmem if you please? https://review.opendev.org/#/c/680300/ | |
| 20:17:15 | mriedem | efried: one of the cores that reviewed that series can hit it | |
| 20:17:21 | mriedem | i wouldn't know if any of it is true or not | |
| 20:17:30 | efried | ack | |
| 20:22:05 | openstackgerrit | Merged openstack/nova master: Refactor pre-live-migration work out of _do_live_migration https://review.opendev.org/641453 | |
| 20:41:35 | melwitt | mriedem: ack, will look | |
| 21:32:42 | openstackgerrit | Eric Fried proposed openstack/nova-specs master: resubmit image metadata prefiltering spec for ussuri https://review.opendev.org/683258 | |
| 21:34:40 | openstackgerrit | Merged openstack/nova-specs master: Spec: Provider config YAML file https://review.opendev.org/680471 | |
| 21:49:12 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Log error when volume validation fails during boot from volume https://review.opendev.org/684140 | |
| 21:49:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional tests for [cinder]/cross_az_attach=False https://review.opendev.org/684141 | |
| 21:49:16 | mriedem | dansmith: ^ i wrote some functional tests for that long-standing cross_az_attach api bug | |
| 21:49:23 | mriedem | rather than rely on devstack wip'ery | |
| 21:50:39 | mriedem | crap, i should actually add the file | |
| 21:50:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional tests for [cinder]/cross_az_attach=False https://review.opendev.org/684141 | |
| 22:02:03 | openstackgerrit | Merged openstack/nova-specs master: resubmit image metadata prefiltering spec for ussuri https://review.opendev.org/683258 | |
| 22:43:48 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Update keypairs in saving an instance object https://review.opendev.org/683043 | |
| #openstack-nova - 2019-09-24 | |||
| 00:22:04 | openstackgerrit | Merged openstack/nova master: Functional reproduction for bug 1844993 https://review.opendev.org/683947 | |
| 00:22:05 | openstack | bug 1844993 in OpenStack Compute (nova) "migrate a server with qos port with compute RPC pinned to 5.1 fails and leaves the qos port in an inconsistent state" [High,In progress] https://launchpad.net/bugs/1844993 - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 00:28:39 | openstackgerrit | Sundar Nadathur proposed openstack/nova-specs master: Re-proposed Nova Cyborg interaction specification. https://review.opendev.org/682637 | |
| 00:28:40 | openstackgerrit | Sundar Nadathur proposed openstack/nova-specs master: Updated Nova-Cyborg interaction spec. https://review.opendev.org/684151 | |
| 02:36:23 | openstackgerrit | Merged openstack/nova master: Add librsvg2* to bindep https://review.opendev.org/683003 | |
| 02:41:12 | openstackgerrit | Takashi NATSUME proposed openstack/nova stable/rocky: Replace glance command with openstack command https://review.opendev.org/637060 | |
| 08:13:41 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject migration with QoS port from conductor if RPC pinned https://review.opendev.org/683948 | |
| 08:49:14 | openstackgerrit | Arthur Dayne proposed openstack/nova master: Fix block disk attachment failure https://review.opendev.org/682772 | |
| 08:49:53 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Remove upgrade specific info from user facing exception text https://review.opendev.org/684253 | |
| 09:05:31 | openstackgerrit | Arthur Dayne proposed openstack/nova master: Fix block disk attachment failure https://review.opendev.org/682772 | |
| 09:16:48 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Move HostNameWeigher to a common fixture https://review.opendev.org/683874 | |
| 09:59:11 | tellesnobrega | hi folks, is there a way from nova to check state of instances and update db if it detects that an instance is not running anymore? | |
| 10:00:13 | tellesnobrega | I saw the reset-state command but wanted to make sure it does what I'm looking for | |
| 11:54:57 | openstackgerrit | Jacek Tomasiak proposed openstack/nova master: Ensure volume filesystems are mounted on resume https://review.opendev.org/684290 | |
| 12:07:16 | gibi | bauzas: left comments and a bug report in https://review.opendev.org/#/c/670112 | |
| 12:11:03 | cdent | gibi: you have a theory about what's going wrong there? | |
| 12:11:47 | gibi | cdent: not yet but I can dig in if needed | |
| 12:12:07 | cdent | no, I was just curious if anything obvious stuck out. and was very happy to see you testing it with real stuff | |
| 12:12:18 | cdent | presumably better functional tests is needed | |
| 12:21:34 | gibi | cdent: I have one commment about context targeting when querying instances from the db that could be the reason | |
| 12:28:57 | gibi | cdent: yeah something with the db access is problematic as the script does not even found the compute node. I guess it querying from the wrong cell db | |
| 12:42:24 | bauzas | gibi: thanks for the comments, as great as usual ;) | |
| 12:42:45 | bauzas | gibi: I'll first write the last functional test and then I'll look at your comments | |
| 12:43:17 | gibi | bauzas: sure. If you need me to re-run the test with more logs or a differnet patch set just let me know | |
| 12:43:41 | gibi | bauzas: or I can provide my devstack conf to test with bandwidth allocation | |
| 12:44:31 | bauzas | gibi: well, why not testing it for bandwith allocations | |
| 12:44:46 | bauzas | gibi: the main issue is that we need also to check child allocations | |
| 13:27:01 | mriedem | artom: https://bugs.launchpad.net/nova/+bug/1845146 this is a known issue right? | |
| 13:27:01 | openstack | Launchpad bug 1845146 in OpenStack Compute (nova) "NUMA aware live migration failed when vCPU pin set" [Undecided,New] | |
| 13:30:49 | artom | mriedem, no, that's the thing that Ya was testing | |
| 13:31:17 | artom | mriedem, http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009608.html | |
| 13:31:55 | artom | I'm gonna look at his logs today, but initial instinct is there's a race somewhere | |
| 13:36:30 | efried | dansmith: Please review this RC candidate with RPC in the title and one +2 from mriedem: https://review.opendev.org/#/c/683948/ | |
| 13:37:05 | dansmith | ah cool, we're on to approving actual bugs during this bug phase? | |
| 13:44:44 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Add reno about nova-api eventlet monkey-patching and rabbitmq https://review.opendev.org/662095 | |
| 13:45:19 | dansmith | mriedem: so on that gibi rpc version patch... | |
| 13:45:55 | dansmith | mriedem: we often gate features on service version, especially when they have API impacts, which is cool, and this is kinda like that | |
| 13:46:33 | dansmith | however, one could take a more RPC-centric world view and say that "if the RPC version allows it I should send it", meaning if they're not set =auto or =stein, they're asking us to send the newer thing | |
| 13:46:53 | dansmith | which could be useful if you're trying to get migrated off the last few computes, or something like that, | |
| 13:47:20 | dansmith | although it could cause problems for *other* calls that might be too new, and/or if you scheduled to an old one but don't have your rpc version pinned properly | |
| 13:47:31 | mriedem | you mean if upgrade_levels=train or =5.3? | |
| 13:47:39 | dansmith | yeah | |