| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-19 | |||
| 18:44:56 | mriedem | looks like something between oslo.db/sqla/pymysql/eventlet switches context and we timeout after some huge amount of time | |
| 18:45:07 | mriedem | nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_models_sync [664.512994s] ... FAILED | |
| 18:45:11 | mriedem | nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_models_sync [39.644814s] ... ok | |
| 18:45:30 | mriedem | clearly we're not just missing some short timeout window | |
| 18:46:09 | dansmith | still waiting for the damn log to load | |
| 18:48:14 | dansmith | wow | |
| 18:48:54 | mriedem | yeah we have a lot of these failures in the gate across master, rocky and queens, | |
| 18:48:55 | mriedem | https://bugs.launchpad.net/cinder/+bug/1793364 | |
| 18:48:55 | openstack | Launchpad bug 1793364 in OpenStack Compute (nova) "mysql db opportunistic unit tests timing out intermittently in the gate (bad thread switch?)" [High,Confirmed] | |
| 18:49:05 | mriedem | nova and cinder b/c they are using the same test fixtures | |
| 18:49:55 | dansmith | hmm | |
| 18:50:30 | openstackgerrit | Eric Fried proposed openstack/nova master: Add contributor guide for upgrade status checks https://review.openstack.org/596902 | |
| 18:50:39 | mriedem | maybe "connection.scalar(select([1]))"? | |
| 18:50:42 | dansmith | does the index complaint surprise you? because it looks like a nonsense name | |
| 18:51:08 | mriedem | well i see this too | |
| 18:51:09 | mriedem | sqlalchemy.exc.ResourceClosedError: This result object does not return rows. It has been closed automatically | |
| 18:51:27 | dansmith | yeah | |
| 18:52:34 | mriedem | http://status.openstack.org/elastic-recheck/data/integrated_gate.html Overall Categorization Rate: 15.4% | |
| 18:52:47 | mriedem | meaning we (openstack) have a shit load of uncategorized failures killing stuff in the gate | |
| 18:52:55 | mriedem | which is why it's taking us days to merge code | |
| 18:57:32 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22%20%20connection.scalar(select(%5B1%5D))'%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d | |
| 19:04:33 | cfriesen | jiteka: I think cold migration should work in mitaka. I think there are some gotchas where resource tracking isn't accurate until the next audit, and maybe there are some resource tracking issues if you revert a resize. | |
| 19:06:13 | tssurya | dansmith: ack, let's skip | |
| 19:06:16 | cfriesen | jiteka: a bit more digging shows that PCI devices were not cold migratable until Newton. | |
| 19:07:13 | mriedem | cfriesen: did you find a specific change that makes that work? | |
| 19:07:42 | mriedem | b/c we could make a note of that on https://docs.openstack.org/nova/latest/admin/pci-passthrough.html | |
| 19:09:34 | cfriesen | mriedem: 5129f48226c I think | |
| 19:10:30 | cfriesen | actually, looks like there are multiple: https://bugs.launchpad.net/nova/+bug/1512880 | |
| 19:10:31 | openstack | Launchpad bug 1512880 in OpenStack Compute (nova) newton "Failed cold migration with SR-IOV" [Medium,Fix released] | |
| 19:10:39 | mriedem | yeah was just going to say that https://review.openstack.org/#/q/topic:bug/1512880+(status:open+OR+status:merged) | |
| 19:11:32 | mriedem | i think it's fair to say it didn't work until newton | |
| 19:28:17 | lucidguy | I believe this bug applies to me, unfortunately I don't know how to resolve, assistance? https://bugs.launchpad.net/tripleo/+bug/1785568 | |
| 19:28:17 | openstack | Launchpad bug 1785568 in OpenStack Compute (nova) "Multiple migration requests for same vm might fail" [Undecided,Incomplete] | |
| 19:29:19 | mriedem | cfriesen: so if you have a minute, could you just update the note at the top of https://docs.openstack.org/nova/latest/admin/pci-passthrough.html about sriov to also mention that cold migration of servers with sriov ports attached didn't work until newton and reference that bug? | |
| 19:42:52 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Return a minimal construct for nova show when a cell is down https://review.openstack.org/591658 | |
| 19:46:23 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Return a minimal construct for nova service-list when a cell is down https://review.openstack.org/584829 | |
| 20:12:14 | openstackgerrit | Merged openstack/nova-specs master: fix spelling mistake https://review.openstack.org/573479 | |
| 20:32:25 | cfriesen | anyone know anything about "bad magic number in nova.conf.crypto" when trying to build nova docs? | |
| 20:33:34 | cfriesen | http://paste.openstack.org/show/730383/ | |
| 20:35:25 | mriedem | i've seen that i think... | |
| 20:37:02 | mriedem | py34 isn't supported... | |
| 20:37:06 | mriedem | not sure if that would be related | |
| 20:37:53 | mriedem | also, | |
| 20:37:57 | mriedem | nova.conf.crypto isn't in master | |
| 20:38:05 | mriedem | is that a starlingx special? | |
| 20:38:29 | mriedem | oh pike | |
| 20:38:29 | mriedem | https://github.com/openstack/nova/blob/stable/pike/nova/conf/crypto.py | |
| 20:39:07 | cfriesen | my regular "python" is 2.7.5. not sure why tox would use 3.4 unless it's searching for whether it's present. I'm on the master branch of upstream nova, though I have checked out pike in this repo. | |
| 20:51:14 | cfriesen | found a nova/conf/crypto.pyc file left over from a previous branch. deleted and retrying | |
| 20:54:33 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Mention SR-IOV cold migration limitation in admin docs https://review.openstack.org/603909 | |
| 20:55:05 | mriedem | cfriesen: huh, i thought we used to have a thing in all tox runs that would clean out pycs | |
| 20:55:13 | mriedem | but i also seem to remember someone removing that years ago | |
| 20:55:28 | mriedem | oh right it's in the base testenv, | |
| 20:55:41 | mriedem | but commands is overridden in the actual testenv:docs run | |
| 20:56:29 | mriedem | cfriesen: oh were you building those docs ^ | |
| 20:56:35 | mriedem | for the thing i just pushed? | |
| 20:58:04 | cfriesen | heh...yep | |
| 20:58:20 | mriedem | sorry, you didn't say anything so i assumed you were busy with something else | |
| 20:58:34 | cfriesen | no worries. | |
| 20:59:51 | openstackgerrit | Mohammed Naser proposed openstack/nova master: Default zero disk flavor to RULE_ADMIN_API in Stein https://review.openstack.org/603910 | |
| 21:00:53 | cfriesen | mriedem: I almost feel like we should use PCI in there instead of SR-IOV, since it's the more generic form. Not all PCI passthrough is SR-IOV, but all SR-IOV is a form of PCI passthrough. | |
| 21:02:24 | openstackgerrit | Mohammed Naser proposed openstack/nova master: Drop migration pre-check error for file_backed_memory https://review.openstack.org/603911 | |
| 21:02:40 | mnaser | mriedem: ^ i thought we were in openstack-nova oops | |
| 21:02:57 | cfriesen | mnaser: you are. :) | |
| 21:03:05 | mnaser | oh no i was talking in #o-operators | |
| 21:03:15 | mnaser | pushing stuffa nd wondering why the hell the bot wasn't notifying | |
| 21:03:38 | cfriesen | that'll do it | |
| 21:04:42 | mriedem | cfriesen: feel free to -1 my change and break my heart | |
| 21:05:12 | mnaser | thats my nova commit for the cycle | |
| 21:05:13 | mriedem | mnaser: i'm guessing you didn't run this through tests yet right? | |
| 21:05:17 | cfriesen | mriedem: I commented on it | |
| 21:05:22 | mnaser | mriedem: nope | |
| 21:05:24 | mnaser | i probably should | |
| 21:07:03 | mnaser | well it looks like tests should pass because the tests use a fixture of using RULE_ADMIN_API and making sure it fails | |
| 21:07:12 | mnaser | so it was testing the then-future case | |
| 21:07:40 | mriedem | mnaser: at least EnforceVolumeBackedForZeroDiskFlavorTestCase in tox -e functional | |
| 21:08:57 | mnaser | mriedem: running | |
| 21:09:05 | mriedem | comments inline | |
| 21:09:58 | mnaser | lemme address after seeing if test breaks | |
| 21:15:57 | openstackgerrit | Mohammed Naser proposed openstack/nova master: Default zero disk flavor to RULE_ADMIN_API in Stein https://review.openstack.org/603910 | |
| 21:19:25 | mriedem | lbragstad: are dashes in policy rule names frowned upon? | |
| 21:19:37 | mriedem | e.g. os_compute_api:servers:cross-cell-resize? | |
| 21:19:42 | mriedem | should be os_compute_api:servers:cross_cell_resize? | |
| 21:19:58 | lbragstad | i don't think so | |
| 21:20:11 | lbragstad | but when i went through a bunch of the projects last week there isn't really a standard | |
| 21:20:20 | mriedem | shocking | |
| 21:20:26 | lbragstad | inoright? | |
| 21:20:46 | lbragstad | so - i guess if we're going to try and do this, i'd like to whack all the moles at once... | |
| 21:20:51 | mriedem | since the prefix is os_compute_api i assume suffix should also be underscores | |
| 21:21:11 | mriedem | oh and we have things like network:attach_external_network and create:zero_disk_flavor so nvm | |
| 21:21:14 | mriedem | answered my own question | |
| 21:21:16 | lbragstad | did you see my note about using the service name? | |
| 21:21:27 | mriedem | no, where? | |
| 21:21:46 | mriedem | i have'nt read the ML thread | |
| 21:22:08 | lbragstad | http://lists.openstack.org/pipermail/openstack-dev/2018-September/134597.html | |
| 21:22:12 | lbragstad | it starts there... | |
| 21:22:24 | lbragstad | sorry if you we waiting to parse the lists later :) | |
| 21:22:35 | mriedem | :( | |
| 21:22:54 | lbragstad | but i'm trying to update that with all context of what i've found across projects (since operators are on that note, too) | |
| 21:24:44 | mriedem | so you'd like to see compute:server:cross_cell_resize | |