| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-02-12 | |||
| 21:51:46 | aspiers | ah it wasn't major, just a method signature change | |
| 21:54:24 | efried | aspiers: I looked at the all-in-one rebase, and you're right, it would be pretty hairy. Thanks for tackling it. | |
| 21:54:45 | aspiers | yw! only 307 commits behind master now :) | |
| 21:55:15 | aspiers | efried: do I understand that for adding an SEV capability to the libvirt driver, capabilities has to be an instance variable rather than a class variable (like for powervm) because the SEV capability is determined dynamically per host? | |
| 21:57:21 | efried | aspiers: I'm pretty sure that's what we figured out for the PowerVM image cache capability, which is based on which disk driver is set up. edmondsw, back me up here. | |
| 21:57:38 | aspiers | yeah, I saw the image cache cap being set at runtime | |
| 21:57:50 | efried | I actually can't remember the exact reason we couldn't use the class variable. | |
| 21:58:37 | efried | or maybe we just thought it was ugly to set PowerVMDriver.capabilities['image_cache'] from the context of 'self'. | |
| 21:58:40 | aspiers | presumably it varies from host to host - does the driver get instantiated once per host? | |
| 21:59:00 | efried | At least once, yes. | |
| 21:59:26 | efried | and it's the "possibly more than once" part that might be what gets you into trouble using a class var. But that doesn't make a lot of sense. | |
| 21:59:48 | efried | Let me go find the review where that happened. | |
| 22:01:03 | efried | aspiers: Ah, it may have just been for testing purposes: https://github.com/openstack/nova-powervm/commit/cc7564549a90a59f2ef27cca14862c33b9a07c25 | |
| 22:02:52 | aspiers | ahhh ok | |
| 22:03:14 | aspiers | rebase finished!! | |
| 22:03:28 | efried | swheet! | |
| 22:03:36 | aspiers | I think there were about 8 commits conflicting in total | |
| 22:03:44 | aspiers | out of ~1000 | |
| 22:03:46 | efried | quick, upload the rebase-only so you have a checkpoint. | |
| 22:03:57 | aspiers | to the same review? | |
| 22:04:00 | efried | absolutely | |
| 22:04:05 | aspiers | OK | |
| 22:05:28 | openstackgerrit | Adam Spiers proposed openstack/nova master: WIP: Convert driver supported capabilities to compute node provider traits https://review.openstack.org/538498 | |
| 22:06:39 | aspiers | efried: there ya go | |
| 22:06:55 | aspiers | who knows, Zuul might even verify it :) | |
| 22:07:11 | cdent | wow, that's like time travel or something | |
| 22:07:42 | efried | aspiers: I'm a fan of doing rebase in one PS and updates in another. Doing an incremental review when rebase+changes are in one PS is suck. | |
| 22:07:54 | aspiers | efried: totally agree | |
| 22:07:55 | efried | especially when the rebase is massive. | |
| 22:08:09 | mriedem | dansmith: comments in the down cell api microversion change https://review.openstack.org/#/c/591657/ - at this point i think i need to pull it into a devstack environment and kick the tires because I think it's being aggressive about processing filters when 'nova list' might send some parameters by default that would otherwise be ignored | |
| 22:08:26 | dansmith | mriedem: okay | |
| 22:08:56 | aspiers | efried: unfortunately in this case diffing between PS7 and my new PS8 is also full of suck, because we're running an ancient Gerrit which can't distinguish between real changes and stuff dragged in by the rebase | |
| 22:09:05 | aspiers | newer Gerrit handles this correctly | |
| 22:09:13 | efried | aspiers: customary to use -W instead of -1 to signify wip with changes on the way. | |
| 22:09:29 | efried | aspiers: Oh, really, there's a better gerrit where that's concerned? That would be pretty amazing. | |
| 22:09:30 | aspiers | efried: I would have if I had W permissions ... | |
| 22:09:40 | efried | aspiers: duh, of course. | |
| 22:09:44 | aspiers | XD | |
| 22:09:59 | aspiers | yes, I remember seeing it demoed at the Gerrit User Summit in London | |
| 22:10:07 | aspiers | I can maybe even find the youtube for you | |
| 22:10:10 | efried | As it stands, I open two tabs, one with old and one with new, and flip-diff 'em. | |
| 22:10:21 | aspiers | yeah, that's better | |
| 22:11:36 | aspiers | efried: https://youtu.be/yWTvUyvP24M?t=963 you can even see my shiny head in the 2nd row ;-) | |
| 22:12:26 | aspiers | although unfortunately we're still on 2.13, not even 2.14 yet let alone 2.15 ... | |
| 22:18:16 | aspiers | OK, time for a very late dinner... | |
| 22:18:23 | aspiers | l8rs | |
| 22:42:25 | edmondsw | efried yep, you found it... sorry, was in a mtg | |
| 22:42:30 | edmondsw | (in a lot of those these days...) | |
| 22:52:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Change nova-next tempest test regex https://review.openstack.org/636459 | |
| 22:52:24 | mriedem | ^ drops a bunch of tempest API tests that we don't care about for nova changes | |
| 22:52:41 | mriedem | hopefully making that job a bit faster and less redundant | |
| 23:08:45 | openstackgerrit | Merged openstack/nova master: Fix legacy-grenade-dsvm-neutron-multinode-live-migration https://review.openstack.org/634962 | |
| 23:10:38 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: API microversion 2.69: Handles Down Cells https://review.openstack.org/579563 | |
| 23:21:21 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: API microversion 2.69: Handles Down Cells https://review.openstack.org/579563 | |
| 23:35:54 | mriedem | dansmith: what would be a simple way to effectively disable/stop the nova_cell1 db without actually stopping mysqld? failing at finding a db stop command | |
| 23:35:57 | mriedem | jaypipes: ^ | |
| 23:36:16 | mriedem | i guess i could eff up the db_connection in the cell1 mapping | |
| 23:36:26 | sean-k-mooney | mriedem: what do you mean by stop? | |
| 23:36:36 | mriedem | i want to make a specific db unavailable | |
| 23:36:40 | sean-k-mooney | as in not allow any new queries/transactins | |
| 23:36:50 | mriedem | i guess | |
| 23:36:57 | mriedem | i want to simulate that the db has crashed | |
| 23:37:00 | mriedem | w/o stopping mysqld | |
| 23:37:09 | dansmith | mriedem: hard to do on a single server and make it real | |
| 23:37:10 | mriedem | since i still need mysql for nova_cell0 and nova_api | |
| 23:37:16 | mriedem | i'll just eff up the db connection then | |
| 23:37:19 | sean-k-mooney | right because you want to allow the other dbs to work | |
| 23:37:21 | dansmith | mriedem: you could revoke privs on the db | |
| 23:37:54 | dansmith | mriedem: or try dropping the tables altogether I guess, but tbh we don't really know what the failure modes will look like I'd guess, and most of the time in reality it would be a connection timeout, | |
| 23:38:02 | dansmith | which will be hard to mimic with one mysql | |
| 23:38:26 | dansmith | one thing you could do, is change the connection string to point to some ip black hole for you | |
| 23:38:39 | dansmith | that would get you the timeout on that connection | |
| 23:38:58 | dansmith | depends on if you're trying to make it happen at runtime, or just start it up unable to reach one of the dbs | |
| 23:40:23 | sean-k-mooney | dansmith: you might be able to do it with ssh forwarding | |
| 23:40:35 | dansmith | do what/ | |
| 23:41:01 | mriedem | yeah i just changed cell1 db connection to mysql+pymysql://root:openstack1@127.0.0.2/nova_cell1?charset=utf8 | |
| 23:41:03 | mriedem | will see what happens | |
| 23:41:16 | sean-k-mooney | e.g. expose the mysql server port locally on anoter port point nova at it and then kill the ssh tunnel e.g. using ssh -L ... | |
| 23:41:21 | dansmith | mriedem: that might not be a black hole for you :) | |
| 23:41:43 | dansmith | mriedem: I would change it to some 172. or 192. or 10. address you definitely can't reach | |
| 23:42:31 | sean-k-mooney | using ssh that way would allow nova to start up normmally and then you can kill the connect when ever you like by killing the ssh tunnel | |
| 23:42:31 | mriedem | oh yeah the connections are cached, | |
| 23:42:33 | mriedem | restarting n-api | |
| 23:43:04 | mriedem | bingo | |
| 23:43:05 | mriedem | Feb 12 23:42:52 downcell devstack@n-api.service[13288]: WARNING oslo_db.sqlalchemy.engines [None req-8002ab53-453c-488e-ae15-04e2a74e7e6f None None] SQL connection failed. 10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on u'192.0.0.1' ([Errno 101] ENETUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) | |
| 23:43:18 | dansmith | right on | |
| 23:43:20 | mriedem | oh goody and it's just spewing in the api logs | |
| 23:43:34 | mriedem | well i guess 10 retries | |
| 23:43:44 | dansmith | sean-k-mooney: he could just do that with some outbound nat to whatever black hole address he chooses | |
| 23:44:02 | mriedem | Feb 12 23:43:49 downcell devstack@n-api.service[13288]: WARNING nova.objects.service [None req-f257afc4-18c9-4989-a500-a4a1f017a322 None None] Cell 0641a06e-4efa-4c25-acac-b92d1f9082b7 did not respond when getting minimum service version | |
| 23:44:02 | mriedem | Feb 12 23:43:49 downcell devstack@n-api.service[13288]: WARNING nova.context [None req-f257afc4-18c9-4989-a500-a4a1f017a322 None None] Timed out waiting for response from cell 0641a06e-4efa-4c25-acac-b92d1f9082b7: CellTimeout: Timeout waiting for response from cell | |
| 23:44:07 | sean-k-mooney | dansmith: ya you can do it with iptables but that is more annoying | |
| 23:44:54 | mriedem | how about i just changed the db connectoin in the db to a bad ip :) | |
| 23:44:56 | sean-k-mooney | with ssh its basically "ssh 127.0.0.1 -L 53306:127.0.0.1:3306" | |
| 23:44:57 | mriedem | much easier | |
| 23:45:31 | sean-k-mooney | mriedem: it is but do you want to be able to kill the connect at an arbitray point or just not have it work | |
| 23:46:08 | mriedem | hmm, nova list with 2.69 hanging...this is probably expected while it's trying to gather results from the down cell db | |
| 23:48:29 | mriedem | i seem to be stuck in an infinite reconnect loop blocking the api response | |
| 23:49:39 | mriedem | and i got a 500 | |
| 23:50:18 | mriedem | oh heh, | |